Simulation Speed

[LukasSchaefer]

Hi all!

I just looked into the speed of the simulation of melting pot substrates and noticed it is much slower than I would expect.
I followed standard installation and use the following package versions (let me know if others could be relevant):

Package                 Version         Editable project location
----------------------- --------------- ---------------------------------------------------------------------
dm-env                  1.5
dm-meltingpot           1.0.1           /home/.../meltingpot
dm-tree                 0.1.6
dmlab2d                 1.0
gym                     0.21.0
numpy                   1.19.5

The code I'm executing is just a minimal execution with random action selection for 1k steps:

from meltingpot.python import substrate
import time

for substrate_name in sorted(substrate.AVAILABLE_SUBSTRATES):
        config = substrate.get_config(substrate_name)
        env = substrate.build(config)
        # _spec_to_space from https://github.com/deepmind/meltingpot/blob/main/examples/rllib/multiagent_wrapper.py
        action_space = _spec_to_space(env.action_spec())
        env.reset()
        start_time = time.time()
        step = 0
        done = False
        while step <= 1000:
            actions = []
            for act_space in action_space:
                actions.append(act_space.sample())
            timestep = env.step(actions)
            if timestep.last():
                env.reset()
            step += 1
        duration = time.time() - start_time
        fps = step/duration
        print(f"{substrate_name}: FPS: {fps} steps/second")

The results I obtained:

Substrate:		 FPS (Steps/ second)
----------------------------------------------------------------------------------------
allelopathic_harvest: 25.75
arena_running_with_scissors_in_the_matrix:	150.66
bach_or_stravinsky_in_the_matrix: 158.30
capture_the_flag: 77.93
chemistry_branched_chain_reaction: 44.19
chemistry_metabolic_cycles:  21.19
chicken_in_the_matrix: 165.99
clean_up: 83.14
collaborative_cooking_impassable: 328.19
collaborative_cooking_passable: 308.44
commons_harvest_closed: 38.12
commons_harvest_open: 52.63
commons_harvest_partnership: 38.86
king_of_the_hill: 75.58
prisoners_dilemma_in_the_matrix:	157.68
pure_coordination_in_the_matrix: 161.73
rationalizable_coordination_in_the_matrix: 153.68
running_with_scissors_in_the_matrix: 399.17
stag_hunt_in_the_matrix: 141.06
territory_open:	136.32
territory_rooms: 115.41

I was a bit surprised to see steps / second of around 50 - 150 for the majority of substrates. In comparison an identical setup for other common multi-agent environments MPE, LBF, RWARE and SMAC:

Gym environment: FPS
------------------------------------------------
SimpleSpeakerListener-v0: 11631.11
SimpleSpread-v0: 2401.27
SimpleAdversary-v0: 6167.93
SimpleTag-v0: 2056.13

smac-2s_vs_1sc: 1336.49
smac-3s5z: 534.54
smac-MMM2: 440.36
smac-3s_vs_5z: 953.64

Foraging-8x8-2p-2f-coop-v1: 7557.26
Foraging-15x15-4p-5f-v1: 3727.38
Foraging-10x10-3p-3f-v1: 5275.40

rware-tiny-2ag-v1: 2236.54
rware-tiny-4ag-v1: 1267.30
rware-small-4ag-v1: 1236.75

While gridworlds like LBF, RWARE and also the MPE environments are fairly simple and I would expect them to be faster than melting pot substrates, I still find it a bit concerning that even SMAC tasks which run the game of StarCraft II run significantly faster than melting pot.

Are these numbers surprising to you? Is there anything that might speed-up the simulation/ are you aware of any bottlenecks?
Given that many substrates seem to require $$10^8$$ to $$10^9$$ numbers of steps of training, these numbers result in quite long training times without access to hundreds of parallel CPUs for simulation.

I would greatly appreciate any thoughts/ suggestions!

[jzleibo]

Hi Lukas, In a word, yes, we are aware of this issue. We are pretty sure that we'll be able to fix the underlying cause. However, we don't know for sure how much performance that will buy. We won't know till we try it.

One of us will reply again to this thread once we have more info to share.

[LukasSchaefer]

I appreciate you coming back to me!
I'll be looking forward to any updates 👍

[duenez]

The latest commit has improved the efficiency of stepping the environment by about 4X. Let us know if this works better for you. Thanks!

[duenez]

I'm closing this issue. If you have any more questions, please don't hesitate to reach out!

[GoingMyWay]

@LukasSchaefer Hi, Lukas, did you get a faster stepping speed? It seems that using a simple core CPU, running 10^9 steps will take 28 days.

[LukasSchaefer]

@GoingMyWay Hey :)
I re-run my speedtest (this is only running action selection with no training) and observed a significant speed-up as mentioned above, but not close to the 4x in the majority of environments. In most substrates, timesteps required about 20-50% less time now compared to the findings I posted above. The code is above, so feel free to test yourself for substrates you are interested in!

[GoingMyWay]

Hi @LukasSchaefer, thanks for the quick sharing. I think to run 10^9 within 3 days, maybe ~40 rollout workers are needed.

[GoingMyWay]

The latest commit has improved the efficiency of stepping the environment by about 4X. Let us know if this works better for you. Thanks!

@duenez, Hi, thanks for updating the version of MeltingPot today. I saw the changelog mentioned the improved speed of simulation. I updated MeltingPot and run the speed test code provided above by @LukasSchaefer on my Intel MacBook Pro (specs shown below). The following are the results:

substrate_name: allelopathic_harvest, FPS: 24.57
substrate_name: arena_running_with_scissors_in_the_matrix, FPS: 154.51
substrate_name: bach_or_stravinsky_in_the_matrix, FPS: 161.54
substrate_name: capture_the_flag, FPS: 93.41
substrate_name: chemistry_branched_chain_reaction, FPS: 39.99
substrate_name: chemistry_metabolic_cycles, FPS: 18.04
substrate_name: chicken_in_the_matrix, FPS: 111.64
substrate_name: clean_up, FPS: 93.10
substrate_name: collaborative_cooking_impassable, FPS: 326.83
substrate_name: collaborative_cooking_passable, FPS: 309.09
substrate_name: commons_harvest_closed, FPS: 51.40
substrate_name: commons_harvest_open, FPS: 59.44
substrate_name: commons_harvest_partnership, FPS: 50.78
substrate_name: king_of_the_hill, FPS: 96.34
substrate_name: prisoners_dilemma_in_the_matrix, FPS: 150.14
substrate_name: pure_coordination_in_the_matrix, FPS: 158.11
substrate_name: rationalizable_coordination_in_the_matrix, FPS: 167.06
substrate_name: running_with_scissors_in_the_matrix, FPS: 484.20
substrate_name: stag_hunt_in_the_matrix, FPS: 162.85
substrate_name: territory_open, FPS: 136.88
substrate_name: territory_rooms, FPS: 126.59

It seems the speed has not been improved much compared with the results posted in #7 (comment). Please correct me if I am wrong.

[GoingMyWay]

The latest commit has improved the efficiency of stepping the environment by about 4X. Let us know if this works better for you. Thanks!

@duenez, Hi, thanks for updating the version of MeltingPot today. I saw the changelog mentioned the improved speed of simulation. I updated MeltingPot and run the speed test code provided above by @LukasSchaefer on my Intel MacBook Pro (specs shown below). The following are the results:

substrate_name: allelopathic_harvest, FPS: 24.57
substrate_name: arena_running_with_scissors_in_the_matrix, FPS: 154.51
substrate_name: bach_or_stravinsky_in_the_matrix, FPS: 161.54
substrate_name: capture_the_flag, FPS: 93.41
substrate_name: chemistry_branched_chain_reaction, FPS: 39.99
substrate_name: chemistry_metabolic_cycles, FPS: 18.04
substrate_name: chicken_in_the_matrix, FPS: 111.64
substrate_name: clean_up, FPS: 93.10
substrate_name: collaborative_cooking_impassable, FPS: 326.83
substrate_name: collaborative_cooking_passable, FPS: 309.09
substrate_name: commons_harvest_closed, FPS: 51.40
substrate_name: commons_harvest_open, FPS: 59.44
substrate_name: commons_harvest_partnership, FPS: 50.78
substrate_name: king_of_the_hill, FPS: 96.34
substrate_name: prisoners_dilemma_in_the_matrix, FPS: 150.14
substrate_name: pure_coordination_in_the_matrix, FPS: 158.11
substrate_name: rationalizable_coordination_in_the_matrix, FPS: 167.06
substrate_name: running_with_scissors_in_the_matrix, FPS: 484.20
substrate_name: stag_hunt_in_the_matrix, FPS: 162.85
substrate_name: territory_open, FPS: 136.88
substrate_name: territory_rooms, FPS: 126.59

It seems the speed has not been improved much compared with the results posted in #7 (comment). Please correct me if I am wrong.

@duenez Dear sir, sorry to ask you agina. Could you please have a test for the speed?

[duenez]

I will take a look. We have another speed update coming soon. I am not sure why the last update didn't improve performance as expected.

[GoingMyWay]

I will take a look. We have another speed update coming soon. I am not sure why the last update didn't improve performance as expected.

That is great👏👏. I am looking forward to running the new version.

[GoingMyWay]

Hi, thanks for the update for improving the simulation speed. After testing, Melting Pot get 3-11x simulation speed improvement:

allelopathic_harvest: FPS: 161.31 steps/second, substrate_index: 1/21
arena_running_with_scissors_in_the_matrix: FPS: 1134.30 steps/second, substrate_index: 2/21
bach_or_stravinsky_in_the_matrix: FPS: 1158.14 steps/second, substrate_index: 3/21
capture_the_flag: FPS: 489.05 steps/second, substrate_index: 4/21
chemistry_branched_chain_reaction: FPS: 118.61 steps/second, substrate_index: 5/21
chemistry_metabolic_cycles: FPS: 47.44 steps/second, substrate_index: 6/21
chicken_in_the_matrix: FPS: 1143.76 steps/second, substrate_index: 7/21
clean_up: FPS: 554.37 steps/second, substrate_index: 8/21
collaborative_cooking_impassable: FPS: 1431.81 steps/second, substrate_index: 9/21
collaborative_cooking_passable: FPS: 1349.03 steps/second, substrate_index: 10/21
commons_harvest_closed: FPS: 170.94 steps/second, substrate_index: 11/21
commons_harvest_open: FPS: 166.71 steps/second, substrate_index: 12/21
commons_harvest_partnership: FPS: 162.62 steps/second, substrate_index: 13/21
king_of_the_hill: FPS: 457.72 steps/second, substrate_index: 14/21
prisoners_dilemma_in_the_matrix: FPS: 1133.82 steps/second, substrate_index: 15/21
pure_coordination_in_the_matrix: FPS: 1064.24 steps/second, substrate_index: 16/21
rationalizable_coordination_in_the_matrix: FPS: 1123.52 steps/second, substrate_index: 17/21
running_with_scissors_in_the_matrix: FPS: 2395.37 steps/second, substrate_index: 18/21
stag_hunt_in_the_matrix: FPS: 1033.93 steps/second, substrate_index: 19/21
territory_open: FPS: 786.97 steps/second, substrate_index: 20/21
territory_rooms: FPS: 342.33 steps/second, substrate_index: 21/21

[duenez]

Glad to hear :)

[GoingMyWay]

Dear @duenez, I read the introduction of substrates chemistry_branched_chain_reaction and chemistry_metabolic_cycles. The two substrates are representative testbeds for reciprocity and convention following MARL research. However, the simulation speed of the two substrates is not fast as in the above table. I understand that you may be occupied with other works and tasks at DeepMind. Could you provide some hints for RL/MARL researchers to improve the speed?

[jzleibo]

We will have new versions of everything coming out soon. Not sure yet if it will help on the speed front or not. It's possible..

[GoingMyWay]

We will have new versions of everything coming out soon. Not sure yet if it will help on the speed front or not. It's possible..

Dear @jzleibo, thank you. I am looking forward to seeing the new release.