Melting Pot Testing

[LukasSchaefer]

Hi all!

I greatly appreciate the effort spent on not just developing the substrates as learning environments, but also spending significant time on the evaluation protocol and testing scenarios with diverse background populations.

However, I am wondering whether there are any plans for scripts of the evaluation protocol?
Some agent models are provided under meltingpot/assets/saved_models and I can see that test scenarios are implemented/ loadable via meltingpot/python/scenario.py (which I assume contains background agents?) but it is still not quite straightforward to identify how a population of trained agents could be evaluated following the protocol described in Section 4.1 of the paper (evaluate for each defined scenario with randomly sampled focal agents sampled from the provided population of trained agents). The closest provided script to such functionality seems to be the meltingpot/python/scenario_test.py script.

Release of any code simplifying a testing protocol as described in the paper would be quite useful!
Apologies if I missed any files/ parts in the documentation describing this procedure.

[jzleibo]

Hi Lukas, thanks for your kind words here. Are you still having trouble getting the testing procedure working?

As for the question about whether the scenarios contain the background populations. The answer is yes. If a particular scenario tests 3 focal agents alongside 5 background agents then it will have 8 players in total in each episode. In this case the num_players attribute of the scenario will be 3 since that's the number of focal agents.

[LukasSchaefer]

Hi, thanks for the pointers.

I think I have it mostly figured out.
Only thing I still wondered about are a bunch of warnings I see when running scenarios. E.g. below for the arena_running_with_scissors_in_the_matrix_0 scenario:

WARNING:absl:Received unexpected argument `[0.]` for path (0, 1), replaced with None.
WARNING:absl:Received unexpected argument `[0.]` for path (0, 2), replaced with None.
WARNING:absl:Received unexpected argument `[1]` for path (0, 3, 'ORIENTATION'), replaced with None.
WARNING:absl:Received unexpected argument `[[1 3]]` for path (0, 3, 'POSITION'), replaced with None.
WARNING:absl:Received unexpected argument `[[0. 0. 0. 0. 1. 0. 0. 0.]]` for path (0, 3, 'agent_slot'), replaced with None.
WARNING:absl:Received unexpected argument `[0.]` for path (0, 1), replaced with None.
WARNING:absl:Received unexpected argument `[0.]` for path (0, 1), replaced with None.
WARNING:absl:Received unexpected argument `[0.]` for path (0, 2), replaced with None.
WARNING:absl:Received unexpected argument `[0.]` for path (0, 2), replaced with None.
WARNING:absl:Received unexpected argument `[0]` for path (0, 3, 'ORIENTATION'), replaced with None.
WARNING:absl:Received unexpected argument `[0.]` for path (0, 1), replaced with None.
WARNING:absl:Received unexpected argument `[0.]` for path (0, 2), replaced with None.
WARNING:absl:Received unexpected argument `[[ 3 22]]` for path (0, 3, 'POSITION'), replaced with None.
WARNING:absl:Received unexpected argument `[2]` for path (0, 3, 'ORIENTATION'), replaced with None.
WARNING:absl:Received unexpected argument `[3]` for path (0, 3, 'ORIENTATION'), replaced with None.
WARNING:absl:Received unexpected argument `[[0. 0. 0. 0. 0. 1. 0. 0.]]` for path (0, 3, 'agent_slot'), replaced with None.
...

I see a bunch a set of these warnings for every step it seems. I believe these might correspond to inputs / outputs around the bots/ background population of agents. What I'm doing is effectively identical to the process described in my performance issue with the only difference that I build scenarios rather than substrates.

Should I be concerned about these warnings or can I safely ignore them?

[duenez]

Hello Lukas,

Yes, I believe those are just warnings due to the discrepancy between training observation spec, and acting observation spec. They are safe to ignore.

Cheers,

Edgar

[duenez]

I'm closing this issue for now. It should be resolved, but do let me know if my understanding is incorrect :)