Skip to content

Scaling up

Example

The full source code of the model can be downloaded here.

This page explains how to scale up a simulation using parallelization using sync-events. We will add new script behaviors called the SyncSender and SyncReceiver. The sender will be part of one submodel and move any entities it receives to receiver in a different submodel. In doing so, we will go through the following topics:

  • How to set up a sync-event.
  • How to send an entity through a sync-event.
  • How to add dependencies between simulators.
  • How to set promises between simulators.
  • How to use a submodel context.
graph LR
  A1@{ shape: stadium, label: "Source" }
  B1@{ shape: rect, label: "Queue" }
  C1@{ shape: rect, label: "Server" }
  D1@{ shape: rect, label: "Sender" }
  A2@{ shape: stadium, label: "Source" }
  B2@{ shape: rect, label: "Queue" }
  C2@{ shape: rect, label: "Server" }
  D2@{ shape: rect, label: "Sender" }

  RECEIVER@{ shape: rect, label: "Receiver" }
  SINK@{ shape: stadium, label: "Sink" }

  subgraph p1 [SubModel 1]
  A1 --> B1;
  B1 --> C1;
  C1 --> D1;
  end
  subgraph p2 [SubModel 2]
  A2 --> B2;
  B2 --> C2;
  C2 --> D2;
  end
  subgraph p3 [SubModel sink]
  RECEIVER
  SINK
  end

  D1 --> RECEIVER;
  D2 --> RECEIVER;
  RECEIVER --> SINK;

Figure 1. Two Source-Queue-Server-Sink submodels who share the same sink (in a third submodel) and run in parallel.

It should be noted that the model we are making is not large enough to require multiple submodels. A single submodel is already capable of simulating many objects. This model is only meant as an example. For guidance on when it makes sense to split a model into submodels in real-world scenarios, see the model splitting page.

Creating the sender and receiver

The sender and the sync-event

The sender will accept one product at a time and use a sync-event to send it to the receiver in a different submodel.

// SyncSender.cs
internal class SyncSenderBehavior : ScriptBehaviorComponent
{
    public ulong SendTime = 10;
    public int TargetSimId;

    public static (SyncSenderBehavior, Entity) Create(bool addTracker = true)
    {
        SubModel subModel = SubModel.Get();

        Entity senderEntity = subModel.CreateEntity("Sender");
        SyncSenderBehavior sender = senderEntity.AddComponent<SyncSenderBehavior>();

        // Channels
        senderEntity.AddComponent<ResourceComponent>();
        Entity senderInputEntity = subModel.CreateEntity(senderEntity, "SenderInput");
        ChannelComponent.AddChannelComponent(senderInputEntity, ChannelType.Input, senderEntity);

        if (addTracker)
            senderEntity.AddComponent<ProductTracker>();

        return (sender, senderInputEntity);
    }

    public override void OnStart()
    {
        Entity input = ConnectedEntity.GetComponent<ResourceComponent>().Value.GetInputChannel(0);
        ChannelComponent.Open(input);
    }

    public override void OnReceive(Entity inputChannel, Entity child)
    {
        SubModel subModel = SubModel.Get();

        // If the entity also has a ProductTracker component, track the products
        if (ConnectedEntity.HasComponent<ProductTracker>())
        {
            ConnectedEntity.GetComponent<ProductTracker>().Value.Seen++;
        }

        Entity input = ConnectedEntity.GetComponent<ResourceComponent>().Value.GetInputChannel(0);
        ChannelComponent.Close(input);

        // Schedule sync-event
        ulong delay = subModel.ApplyModelPrecision(SendTime);
        SendEntitySyncEvent syncEventData = new();
        syncEventData.MoveEntity = child;
        syncEventData.SenderEntity = ConnectedEntity;
        EventScheduler.ScheduleSyncEvent<SendEntitySyncEvent>(delay, TargetSimId, syncEventData);

    }
}

As you can see, the sender only needs to implement the OnStart and OnEntered callbacks. Just like in the other script behaviors, OnStart is used to open the channel as soon as the simulation starts.

When the sender receives an entity, it first updates the tracker. Next it closes the input so no new products are received, and lastly it schedules the sync-event. A sync-event is scheduled using a sync-event type, a delay, a target simulator, and the event data. The delay works the same as with local events and the target simulator is the simulator to which the sync-event is scheduled and thus data can be sent.

The sync-event type determines the behavior of the sync-event. Below is the implementation of the SendEntitySyncEvent. It moves a product from the sender submodel to the target submodel and sets the sender's input-channel as open so it can receive a new product.

// SyncSender.cs
[StructLayout(LayoutKind.Sequential)]
public partial struct SendEntitySyncEvent : ISyncEvent<SendEntitySyncEvent>
{
    public static string Name => "Send entity to other simulator";

    private MoveEntitySyncEvent moveEvent;

    public Entity MoveEntity
    {
        set
        {
            // Pass along the entity to the internal sync-event
            moveEvent.EntityInFlight = value;
        }
    }
    // This entity only exists on the sender submodel!
    public Entity SenderEntity;

    public void OnSenderSide()
    {
        Logger.Debug($"Sending {moveEvent.EntityInFlight.GetName()} through a sync-event");
        moveEvent.OnSenderSide();

        // Open the input channel of the sender again so it can accept more products
        Entity input = SenderEntity.GetComponent<ResourceComponent>().Value.GetInputChannel(0);
        ChannelComponent.Open(input);
    }

    public void OnTargetSide()
    {
        moveEvent.OnTargetSide();
        Logger.Debug($"Receiving {moveEvent.EntityInFlight.GetName()} through a sync-event");

        // Move the received entity to the sink
        SubModel subModel = SubModel.Get();
        var context = subModel.GetSubModelContext<SinkContext>();
        subModel.UpdateParentOnEntity(moveEvent.EntityInFlight, context.ReceiverEntity);
    }
}

Let's look at the implementation of the SendEntitySyncEvent sync-event.

  • At the top, you can see the sync-event has a name. The purpose of the sync-event name is to be able to easily identify sync-events when debugging a model.
  • Below it, you see we are using the pre-existing MoveEntitySyncEvent sync-event as moveEvent. This pre-built sync-event is meant to conveniently move entities between submodels, if that is all you require. We want to do a bit more, so we wrap our sync-event around it to add the additional functionality we need.
  • Next, you see the MoveEntity variable. This wraps the EntityInFlight variable in the moveEvent so we can easily select which entity to move.
  • The SenderEntity is stored to keep track of which sender scheduled this sync-event. We use it inside the OnSenderSide function to open its input-channel again.

The OnSenderSide function is called in the sender simulator when the sync-event is scheduled. On the target side there is the OnTargetSide function to define the behavior of the sync-event when it is executed in the target simulator. In this case, we first call the OnSenderSide and OnTargetSide functions of MoveEntitySyncEvent to perform the moving of the entity. Additionally, on the sender side we open the input-channel of the sender so it can accept more products. On the target side, we move the received entity to the SyncReceiver using UpdateParentOnEntity. The receiver entity is stored in the submodel context of type SinkContext. This will be explained and implemented in the setting up the submodel context section.

Do not mix submodel data!

It is very important not to mix data from different submodels. For example, this sync-event stores the SenderEntity. This entity only exists on the sender submodel. If the target submodel tries to use this entity, it will either:

  • Use an invalid entity ID and cause a (potentially fatal) error.
  • Use the entity with the same entity ID in the target submodel (which is not the sender), leading to an incorrect simulation.

A safer approach that could have been used for the sync-event above is to not store the sender entity and always re-acquire it instead:

public void OnSenderSide()
{
    Logger.Debug($"Sending {moveEvent.EntityInFlight.GetName()} through a sync-event");
    moveEvent.OnSenderSide();

    // Open the input-channel of the sender again so it can accept more products
    SubModel subModel = SubModel.Get();
    Entity foundEntity = subModel.FindEntity("SenderEntity");
    foundEntity.GetComponent<ResourceComponent>().Value.GetInputChannel(0);
    ChannelComponent.Open(input);
}

However, this has the disadvantage that the sync-event becomes less flexible: the OnSenderSide function needs to know what the entity is going to be called. This becomes especially problematic when a submodel has multiple senders, each with a different name. Additionally, the entity needs to be searched for, which is less efficient than storing the entity ID.

The receiver

The SyncReceiver receives entities from the SyncSender. Because the entity is received via a sync-event and not channels, we will use OnEntered again instead of OnReceived. In OnEntered, we first check if the received entity is not the output-channel of the receiver. This is necessary because the OnEntered callback is triggered for each child entity that is added to the receiver entity, which includes the channel entity when it's created by CreateEntity in the Create function of SyncReceiver. After that, the tracking is performed, just like in all other script behaviors. Lastly, the received entity is send through the output-channel of the receiver.

// SyncReceiver.cs
internal class SyncReceiver : ScriptBehaviorComponent
{
    private Entity OutputChannel = CEntity.InvalidEntity();

    public static (SyncReceiver, Entity) Create(string name = "Receiver", bool addTracker = true)
    {
        SubModel subModel = SubModel.Get();

        Entity receiverEntity = subModel.CreateEntity(name);
        SyncReceiver receiver = receiverEntity.AddComponent<SyncReceiver>();

        // Channels
        receiverEntity.AddComponent<ResourceComponent>();
        Entity receiverOutputEntity = subModel.CreateEntity(receiverEntity, $"{name}-Output");
        ChannelComponent.AddChannelComponent(receiverOutputEntity, ChannelType.Output, receiverEntity);

        if (addTracker)
            receiverEntity.AddComponent<ProductTracker>();

        return (receiver, receiverOutputEntity);
    }

    public override void OnStart()
    {
        OutputChannel = ConnectedEntity.GetComponent<ResourceComponent>().Value.GetOutputChannel(0);
        ChannelComponent.Open(OutputChannel);
    }

    public override void OnEntered(Entity newChild)
    {
        // When the output-channel is created, it also triggers this function. Ignore it
        if (newChild == OutputChannel || OutputChannel == CEntity.InvalidEntity())
            return;

        // If the entity also has a ProductTracker component, track the products
        if (ConnectedEntity.HasComponent<ProductTracker>())
        {
            ConnectedEntity.GetComponent<ProductTracker>().Value.Seen++;
        }

        Entity output = ConnectedEntity.GetComponent<ResourceComponent>().Value.GetOutputChannel(0);
        if (ChannelComponent.IsReady(output))
        {
            ChannelComponent.Send(output, newChild);
        }
    }
}

Setting up the submodel context

A submodel context is a piece of data you can attach to a submodel. Each simulator can have multiple submodel contexts, as long as they are each of a different type. For this model, we use a submodel context to store the sink entity. The reason we do this, is because the submodel the sender resides in does not know about entities in the sink's submodel. The submodel context allows the sync-event to easily find the sink entity and send the products to it.

// Program.cs
internal class SinkContext
{
    public Entity SinkEntity;
}

As you can see, a submodel context is just a class with some data. To add the context to the submodel, use the function AddSubModelContext. We will do this later when we set up the sink's submodel.

Updating the model setup

We will now update the model we made previously. As shown in Figure 1, we will move the sink to a separate submodel and create two Source-Queue-Server submodels that feeds into the sink submodel.

We want the main method of the program to become like this:

// Program.cs
static void Main(string[] args)
{
    ERS.Initialize();

    Logger.SetLogLevel(LogLevel.Info);

    // Register the custom components
    ComponentRegistry<SourceBehavior>.Register();
    ComponentRegistry<QueueBehavior>.Register();
    ComponentRegistry<ServerBehavior>.Register();
    ComponentRegistry<SinkBehavior>.Register();
    ComponentRegistry<SenderBehavior>.Register();
    ComponentRegistry<Product>.Register();
    ComponentRegistry<ProductTracker>.Register();

    // Register the local events
    LocalEventRegistry<GenerateProductEvent>.Register();
    LocalEventRegistry<ServerProcessEvent>.Register();

    // Register the sync-events
    SyncEventRegistry<SendEntitySyncEvent>.Register();

    ModelContainer modelContainer = ModelContainer.Create();
    modelContainer.Precision = 1_000_000;
    modelContainer.Seed = 1;

    Simulator sinkSim = CreateSinkSubModel(modelContainer);
    Simulator sim1 = AddSubModel(modelContainer, sinkSim, "Sim1");
    Simulator sim2 = AddSubModel(modelContainer, sinkSim, "Sim2");

    // Set the dependencies between simulators
    modelContainer.AddSimulatorDependency(sim1, sinkSim);
    modelContainer.AddSimulatorDependency(sim2, sinkSim);

    // Set the promises between simulators
    sim1.EnterSubModel();
    EventScheduler.SetPromise(sinkSim.ID, 5 * modelContainer.Precision);
    sim1.ExitSubModel();
    sim2.EnterSubModel();
    EventScheduler.SetPromise(sinkSim.ID, 5 * modelContainer.Precision);
    sim2.ExitSubModel();

    ulong endTime = 3600 * modelContainer.Precision;
    while (modelContainer.CurrentTime < endTime)
    {
        modelContainer.Update(modelContainer.Precision);
    }

    LogResult(sim1);
    LogResult(sim2);
    LogResult(sinkSim);

    ERS.Uninitialize();
}

First of all, we register the sync event, just like we do with local events. Instead of using the LocalEventRegistry, we use the SyncEventRegistry.

The CreateSinkSubModel and AddSubModel methods each create a new simulator and add the right entities and components. We pass the modelContainer to each of them so they can create the simulators on it. The sink's simulator is passed to the other two submodel so we can set the TargetSimId in the senders. We will look at their implementations later.

Notice how we define both dependencies and promises between simulators. The dependencies define which simulators can send sync-events to which other simulators. The promises define what the minimum time is between scheduling an event and executing it on the target simulator (the minimum delay time). When defining a dependency, the promise is implicitly set to 0. At the same time, promises cannot be set without defining a dependency between simulators. Higher values for promises in general allow ERS to run more in parallel, resulting in a faster model. For this model, we have set the promises to an arbitrary value of 5 simulation seconds (taking the precision into account: \(5 \times 10^6\) time units). For more information about dependencies and promises, please see their respective sections in the core documentation.

The LogResult function is a convenience function to log the results per submodel. It's the view we defined earlier. The only change is that it's now using the simulator given to the function, instead of using the only simulator we had before. We also added another log at the start to indicate which simulator's we're viewing.

// Program.cs
static void LogResult(in Simulator sim)
{
    Logger.Info($"----------[{sim.Name}]----------");

    // Log how many products each of the simulation objects saw
    sim.EnterSubModel();
    SubModel subModel = SubModel.Get();

    var trackerView = subModel.GetView<ProductTracker>([]);
    while (trackerView.Next())
    {
        Entity entity = trackerView.GetEntity();
        var tracker = trackerView.GetComponent<ProductTracker>();
        Logger.Info($"{entity.GetName()} saw {tracker.Value.Seen} products");
    }
    trackerView.Dispose();

    sim.ExitSubModel();
}

As you will notice later when we run the model, this function is very flexible: the view remains the same, but it automatically picks up any compatible entities on any submodel.

Creating the first two submodels

Let's first create the AddSubModel method by re-using the existing model. We take the exact same setup, except we replace the sink with the sender. As you can see below, the setup is almost exactly the same as before.

// Program.cs
static Simulator AddSubModel(ModelContainer modelContainer, Simulator sinkSim, string name)
{
    Simulator sim = modelContainer.AddSimulator(name, SimulatorType.DiscreteEvent);
    sim.EnterSubModel();

    // Create the simulation objects
    (_, Entity sourceOutputEntity) = SourceBehavior.Create();
    (_, Entity queueInputEntity, Entity queueOutputEntity) = QueueBehavior.Create();
    (_, Entity serverInputEntity, Entity serverOutputEntity) = ServerBehavior.Create();
    (SyncSenderBehavior sender, Entity senderInputEntity) = SyncSenderBehavior.Create();

    // Configure simulation objects
    sender.TargetSimId = sinkSim.ID;

    // Connect the simulation objects
    ChannelComponent.Connect(sourceOutputEntity, queueInputEntity);
    ChannelComponent.Connect(queueOutputEntity, serverInputEntity);
    ChannelComponent.Connect(serverOutputEntity, senderInputEntity);

    sim.ExitSubModel();

    return sim;
}

The sender entity is created just like the others. The only difference is that we set the TargetSimId to the sink's simulator's ID.

Creating the sink submodel

The sink's submodel is created the same way as the other submodel.

// Program.cs
static Simulator CreateSinkSubModel(ModelContainer modelContainer)
{
    Simulator sim = modelContainer.AddSimulator("SinkSim", SimulatorType.DiscreteEvent);
    sim.EnterSubModel();

    SubModel subModel = SubModel.Get();

    // Create simulation objects
    (SyncReceiver receiver, Entity receiverOutputEntity) = SyncReceiver.Create();
    (_, Entity sinkInputEntity) = SinkBehavior.Create();

    // Connect simulation objects
    ChannelComponent.Connect(receiverOutputEntity, sinkInputEntity);

    var context = subModel.AddSubModelContext<SinkContext>();
    context.ReceiverEntity = receiver.ConnectedEntity;

    sim.ExitSubModel();
    return sim;
}

We are finally adding the submodel context. As you can see, adding a submodel context works similarly to the components via the AddSubModelContext method. It also returns the added context, allowing us to set the SinkEntity to the right value. As you saw in the sync-event definition, the context can be retrieved using the GetSubModelContext method.

Running multiple submodels

We've changed quite a bit in the Program.cs file, so here is the full file by the end of this page for clarity:

// Program.cs
using Ers;

namespace ScalingUp
{
    internal class SinkContext
    {
        public Entity ReceiverEntity;
    }

    internal class Program
    {
        static Simulator CreateSinkSubModel(ModelContainer modelContainer)
        {
            Simulator sim = modelContainer.AddSimulator("SinkSim", SimulatorType.DiscreteEvent);
            sim.EnterSubModel();

            SubModel subModel = SubModel.Get();

            // Create simulation objects
            (SyncReceiver receiver, Entity receiverOutputEntity) = SyncReceiver.Create();
            (_, Entity sinkInputEntity) = SinkBehavior.Create();

            // Connect simulation objects
            ChannelComponent.Connect(receiverOutputEntity, sinkInputEntity);

            var context = subModel.AddSubModelContext<SinkContext>();
            context.ReceiverEntity = receiver.ConnectedEntity;

            sim.ExitSubModel();
            return sim;
        }

        static Simulator AddSubModel(ModelContainer modelContainer, Simulator sinkSim, string name)
        {
            Simulator sim = modelContainer.AddSimulator(name, SimulatorType.DiscreteEvent);
            sim.EnterSubModel();

            // Create the simulation objects
            (_, Entity sourceOutputEntity) = SourceBehavior.Create();
            (_, Entity queueInputEntity, Entity queueOutputEntity) = QueueBehavior.Create();
            (_, Entity serverInputEntity, Entity serverOutputEntity) = ServerBehavior.Create();
            (SyncSenderBehavior sender, Entity senderInputEntity) = SyncSenderBehavior.Create();

            // Configure simulation objects
            sender.TargetSimId = sinkSim.ID;

            // Connect the simulation objects
            ChannelComponent.Connect(sourceOutputEntity, queueInputEntity);
            ChannelComponent.Connect(queueOutputEntity, serverInputEntity);
            ChannelComponent.Connect(serverOutputEntity, senderInputEntity);

            sim.ExitSubModel();

            return sim;
        }

        static void LogResult(in Simulator sim)
        {
            Logger.Info($"----------[{sim.Name}]----------");

            // Log how many products each of the simulation objects saw
            sim.EnterSubModel();
            SubModel subModel = SubModel.Get();

            var trackerView = subModel.GetView<ProductTracker>([]);
            while (trackerView.Next())
            {
                Entity entity = trackerView.GetEntity();
                var tracker = trackerView.GetComponent<ProductTracker>();
                Logger.Info($"{entity.GetName()} saw {tracker.Value.Seen} products");
            }
            trackerView.Dispose();

            sim.ExitSubModel();
        }

        static void Main(string[] args)
        {
            ERS.Initialize();

            Logger.SetLogLevel(LogLevel.Info);

            // Register the custom components
            ComponentRegistry<SourceBehavior>.Register();
            ComponentRegistry<QueueBehavior>.Register();
            ComponentRegistry<ServerBehavior>.Register();
            ComponentRegistry<SinkBehavior>.Register();
            ComponentRegistry<SyncSenderBehavior>.Register();
            ComponentRegistry<SyncReceiver>.Register();
            ComponentRegistry<Product>.Register();
            ComponentRegistry<ProductTracker>.Register();

            // Register the local events
            LocalEventRegistry<GenerateProductEvent>.Register();
            LocalEventRegistry<ServerProcessEvent>.Register();

            // Register the sync events
            SyncEventRegistry<SendEntitySyncEvent>.Register();

            ModelContainer modelContainer = ModelContainer.Create();
            modelContainer.Precision = 1_000_000;
            modelContainer.Seed = 1;

            Simulator sinkSim = CreateSinkSubModel(modelContainer);
            Simulator sim1 = AddSubModel(modelContainer, sinkSim, "Sim1");
            Simulator sim2 = AddSubModel(modelContainer, sinkSim, "Sim2");

            // Set the dependencies between simulators
            modelContainer.AddSimulatorDependency(sim1, sinkSim);
            modelContainer.AddSimulatorDependency(sim2, sinkSim);

            // Set the promises between simulators
            sim1.EnterSubModel();
            EventScheduler.SetPromise(sinkSim.ID, 5 * modelContainer.Precision);
            sim1.ExitSubModel();
            sim2.EnterSubModel();
            EventScheduler.SetPromise(sinkSim.ID, 5 * modelContainer.Precision);
            sim2.ExitSubModel();

            ulong endTime = 3600 * modelContainer.Precision;
            while (modelContainer.CurrentTime < endTime)
            {
                modelContainer.Update(modelContainer.Precision);
            }

            LogResult(sim1);
            LogResult(sim2);
            LogResult(sinkSim);

            ERS.Uninitialize();
        }
    }
}

If we now run the model, the output should log how many products each object that has a channel component saw, for each submodel. As you can see, the LogResult function works as expected and we have the results of all submodels. Of course, if you only want to see the results of the sink's submodel, you only have to call LogResult for that submodel.

[I] Thread [49380] Starting Enterprise Resource Simulator Core
[I] Thread [49380] Version 0.7.0 | Build Date: 2026-04-23 15:13:41
[I] Thread [49380] Reading initial settings file
[I] Thread [49380] [Info] is new LogLevel!
[I] Thread [49380] [Info] is new LogLevel!
[I] Thread [49380] ----------[Sim1]----------
[I] Thread [49380] Source saw 238 products
[I] Thread [49380] Queue saw 237 products
[I] Thread [49380] Server saw 232 products
[I] Thread [49380] Sender saw 231 products
[I] Thread [49380] ----------[Sim2]----------
[I] Thread [49380] Source saw 241 products
[I] Thread [49380] Queue saw 240 products
[I] Thread [49380] Server saw 235 products
[I] Thread [49380] Sender saw 234 products
[I] Thread [49380] ----------[SinkSim]----------
[I] Thread [49380] Receiver saw 463 products
[I] Thread [49380] Sink saw 463 products
[I] Thread [49380] Waiting for JobSystem to finish remaining tasks, then shut down
[I] Thread [49380] Exiting JobSystem
[I] Thread [49380] Closing Enterprise Resource Simulator Core

Next steps

This is the end of the getting started tutorial for ERS. If you want to learn more about the features of ERS, you can browse the Core pages of this website, where all features covered in this tutorial and more are covered in more depth.

Was this page helpful?