Sol env01

[snowgoon88]

some explanation about the 'switcher' model

I define an "agent" with a subsumption architecture using 3 "behaviors". The agent follows the outer wall turning in an anti-clockwise direction and avoids obstacles too close on its left. When its energy rises up, it is a kind of positive reward signal for the agent that "memorize" it so as to trigger an "early" turn left in the middle corridor. At the end of this "turning left in the middle corridor", the agent silences the reward signal and its memorization.

Oh, and I'm using "ReLu" activation for neurons, easier to work with (on/off) for handcrafting weights.

This "handcrafted" agent could be streamlined, its behavior is not very robust and it sometimes get stuck. But the desired general behavior is there.

behavior : "away from right"

When input from the rightmost sensors is below a threshold, turn left (positive output).

    # TurnLeft (away from SensorRight)
    W_in[1, i_right] = 1.0
    W_in[1, i_bias] = -0.79
    W_out[0, 1] = 100

behavior : "avoid close things on the left"

Sometimes, the agents comes to close to a left wall, so it avoids it by turning right (negative output when input of the leftmost sensor is above a threshold)

    # TurnRight (avoid close SensorLeft)
    W_in[i_turn+1, i_left] = 1.0
    W_in[i_turn+1, i_bias] = -0.88
    W_out[0, i_turn+1] = -50

behavior : "when rewarded, take the first left"

First, the agents needs to create a "reward" signal, based on it energy variation. Energy variation is computed as inputEnergy(t) - inputEnergy(t-1), and cumulated if positive (being the main motivation for using ReLu activation function). BUT, as the initial difference is high (as the inputEnergy(0-1) is 0), I need to filter out "large" energy bumps.

Then, the reward signal is "memorized" (and slightly amplified) in X[i_filter+2,0], this is a signal for initiating the "turn into the mid corridor".

Not very proud of this part that seems a bit over complicated.

    # detect NRJ increase.
    W_in[i_nrj, i_nrj] = 1.0
    W[i_nrj+1, i_nrj] = 1.0
    W[i_nrj+2, i_nrj] = 1.0
    W[i_nrj+2, i_nrj+1] = - 1.0
    # need a band-filter so as to only detect "small" NRJ bumps
    # (and not the starting bump with value '1')
    W[i_filter, i_nrj+2] = 1.0
    W[i_filter, i_bias]  = -0.004
    W[i_filter+1, i_nrj+2] = 1.0
    # i_filter+2 accumulate and "memorise" reward increase
    W[i_filter+2, i_filter]   = -2.0
    W[i_filter+2, i_filter+1] = 1.0
    W[i_filter+2, i_filter+2] = 1.05
    W[i_filter+2, i_turn_mid+4] = -1000.0  # shut off when end_midturn
    # i_filter+3 is a NOT(i_filter+2)
    W[i_filter+3, i_filter+2] = -1.0
    W[i_filter+3, i_bias] = +0.07

To trigger the turn in the middle corridor (only when the reward signal is high enough), the agent kind of detects when the leftmost sensor is low (below threshold) for sometime. Then it turns right until a centerSensor is low enough (meaning the agents faces a far away wall). When "turning into the mid corridor" ends, the agents shuts off the reward accumulation (X[i_filter+2, 0]) and the signal that triggered the "mid_turn" (X[i_turn_mid+1]).

    # "mid_turn" are started when a far away obstacle on i_left has been seen
    # enough time since some time (hence the * 1.05 feedback on i_turn_mid+1)
    # accumulate i_left < 0.40 when rewarded
    W_in[i_turn_mid, i_left] = -1.0
    W_in[i_turn_mid, i_bias] = 0.45
    W[i_turn_mid, i_filter+3] = -10 # do not detect if no reward received yet
    # accumulate diff
    W[i_turn_mid+1, i_turn_mid] = 1.6
    W[i_turn_mid+1, i_turn_mid+1] = 1.07
    W[i_turn_mid+1, i_turn_mid+4] = -1000  # shut off at end of mid_turn
    # check when to start (threshold i_turn_mid+2)
    W[i_turn_mid+2, i_turn_mid+1] = 1.0
    W[i_turn_mid+2, i_bias] = -1.3

    # when to stop "mid_turn"
    # need to know when NOT(mid_turn)
    W[i_turn_mid+3, i_turn_mid+2] = -1.0
    W_in[i_turn_mid+3, i_bias] = 1.3
    # detect if center_right < 43 AND turning with i_turn_mid+2 (i.e i_turn_mid+2 > 0)
    W_in[i_turn_mid+4, i_center_right] = -1
    W_in[i_turn_mid+4, i_bias] = 0.43
    W[i_turn_mid+4, i_turn_mid+3] = -1   # no while NOT mid_turn

    # finally, apply to output
    W_out[0, i_turn_mid+2] = 100

[vforch]

Here's a slightly simpler (and super hacky) version (12 instead of 20 active nodes) of this idea that reaches an optimal score of ~14.7.

    """Optimal heuristic driving always left and going through the 'tunnel' once it encountered reward."""
    bot = Bot()
    max_energy= 1.030
    
    # network hyperparameters
    leak = 0.95
    def act(x):
        x = np.tanh(x)
        return np.where(x > 0, x, 0)
    
    # model size
    n_cam = bot.camera.resolution
    n_inp = n_cam + 3
    n_rec = 1000 
    
    n_min = n_cam - 1  # only using every n_min'th sensor node
    
    model_values = 1, 0.77, 0.79, 0.98, 0.82, 0.01, 0.1
        
    # input parameters
    exc_input = model_values[0]
    thresh_inp = model_values[1]
    
    thresh_inp_2 = 2 * model_values[2]
    
    # energy triggers
    energy_level_1 = max_energy * model_values[3]
    energy_level_2 = max_energy * model_values[4]
    
    # steering parameters
    bias_left_static = model_values[5]
    bias_left_triggered = model_values[6]  
    
    # construct the model
    W_in = np.zeros((n_rec, n_inp))
    W = np.zeros((n_rec, n_rec))
    W_out = np.zeros(n_rec)
    
    model = W_in, W, W_out, 0, leak, act, identity

    # input weights for connecting sensor to steering pop
    for i in range(0, n_cam, n_min):  W_in[i, i] = exc_input
    W_in[:n_cam:n_min, -1] = -thresh_inp
    
    # 2nd pop for opposite steering behaviour (draws agent towards walls)
    for i in range(0, n_cam, n_min):  W_in[i+n_cam, i] = exc_input
    W_in[n_cam:2*n_cam:n_min, -1] = -thresh_inp_2
    
    # reward sensor - input
    W_in[2*n_cam+1, n_cam+1] = 1
    W_in[2*n_cam+2, n_cam+1] = -1
    # energy threshold - fast reward (triggers on left reward)
    W_in[2*n_cam+1, n_cam+2] = -energy_level_1
    # energy threshold - delayed reward (triggers on right reward)
    W_in[2*n_cam+2, n_cam+2] = energy_level_2
    
    # persistent activity once triggered
    W[2*n_cam+1, 2*n_cam+1] = 1000
    W[2*n_cam+2, 2*n_cam+2] = 1000
    # inhibition of delayed reward trigger
    W[2*n_cam+2, 2*n_cam+1] = -1000

    # activation of second steering pop
    W[n_cam:2*n_cam:n_min, 2*n_cam+1] = 1
    W[n_cam:2*n_cam:n_min, 2*n_cam+2] = 1

    # constant left bias
    W_in[2*n_cam+3, n_cam+2] = 1

    # output weights
    W_out[:n_cam:n_min] = n_min * np.linspace(-1, 1, len(W_out[:n_cam:n_min]))
    W_out[n_cam:2*n_cam:n_min] = bias_left_triggered * n_min * np.linspace(1, -1, len(W_out[:n_cam:n_min]))
    W_out[2*n_cam+1] = bias_left_triggered
    W_out[2*n_cam+2] = bias_left_triggered
    W_out[2*n_cam+3] = bias_left_static
    
    yield model```

[rougier]

Thanks, let me test it. And since @vforch proposed an improvement, you can you two submit another entry with the modification.

[rougier]

I create the new entry. Impressive results.
Now I've to modify the second challenge just because of your almost perfect results.

[snowgoon88]

Nicely done @vforch , I'm also very impressed.

Take your time before the next challenge, @rougier , I will NOT have so much free time now that the academic year started again...