Assets AI That Adapts To & Overcomes the Player (Master's Thesis Project)

Hi Guys!

For a few months now, I have been working on a project, in which the goal was to develop an Artificial Intelligence (AI) that would learn as the game was played and get better at what it did. By doing so I was aiming for AI that would dynamically get more difficult and keep the player engaged.



To achieve this I use an algorithm called NEAT (NeuroEvolution of Augmenting Topologies). What this basically does is to evolve Artificial Neural Networks(Brains!) using the Evolution Theory (+ additional stuff to make it work better).
I use a modified version of this algorithm, called rtNEAT (real time NEAT...that evolves in 'real time'), which I then further modified to only evolve Neural Networks of recently killed Enemy AI Agents.

I have one month left for the deadline of this project, so I will share some data in this thread as I write my paper.

Additionally, I will need your help at some point! You will be trying the project and kill some AI Agents and hopefully see them overcome and kill you.
Get the project here: https://www.dropbox.com/s/nzzfx9d4wl46eun/INEATv1.zip?dl=0
(I will be able to use data from your playthroughs in my Master's Thesis paper).
(If you are interested, please let me know in this thread).

If you have any questions, I will be happy to answer (or atleast try to )

- Muhsin Kaymak

 

I'm more interested in the source code though lol

 

The source code of NEAT coded for Unity is freely available here:
https://github.com/lordjesus/UnityNEAT
It uses generational evolution.

I used that version and modified it for my needs. I will be sure to upload my code somewhere once I finished my master's thesis( and if I have the time to clean up the code)

 

Thank you Lord Jesus!

 

First Data (or the lack of results)
Hi Guys! here follows few results of the Experiments:

For this experiment, I hardcoded a 'Player' AI, that goes around killing Evolving AI Agents.
The Player AI went around and eliminated 144 populations, which each contain 15 Agents, so the Player AI killed in total 2160 AI Agents in this run.

The Evolving AI Agents are encouraged to improve their accuracy and 'hit on target' rate, using a Fitness function.

The first graph shows each population's average 'hit on target' rate:


The second graph shows each population's average accuracy:



By looking at the graph, we can see that there is a slight improvement in accuracy and even less improvement on 'hits on target'.

There can be few reasons for this:

1. The population size is too small.
2. The Agents didn't have enough data for significant improvements.
3. Because of the AI Agents being evaluated and reproduced as they are killed (in contrast to generational evolution, where all Agents in a population are evaluated and reproduced at the same time).
4. The fitness function is not effective enough.

additionally, the graph looks very 'noisy', while it should be more smooth to indicate an effective evolution.
This might be because of each population being divided by rooms and these rooms each having different dimensions and objects in them, resulting in different behaviours for both the AI Agents and the Player AI.

I will continue my investigation and collect more data. What are your thoughts?

 

Interesting, what are you using as your fitness function?

 

Pretty simple:

Code (CSharp):

1.  
2. if(op.globalBestHit > 0)
3. {
4.       hitFitness = ((float)hitsOnTarget / (float)op.globalBestHit);
5. }
6. if(op.globalBestAccuracy > 0)
7. {
8.       accFitness = (((float)hitsOnTarget/(float)bulletsFired) / (float)op.globalBestAccuracy);
9. }
10.  
11. fitness = (accFitness+hitFitness) / 2;
12.  

I first get the 'hit on target' fitness and the accuracy fitness.
Then I add them together and normalize.

 

Want to help me collect data?

Hi Guys,
If you want to test out the project and help me collect data for my Master Thesis, then you can download the game from here:
https://www.dropbox.com/s/nzzfx9d4wl46eun/INEATv1.zip?dl=0

The Agents should start differently each time. Sometimes they don't use their weapon, sometimes their aim is way off and sometimes they just stand there waiting to die.
The goal is to see if they improve as you kill them.
After killing a bunch, you should start seeing Agents that get better at aiming or avoiding your bullets.

The game will automatically upload the agents fitness, so you need to have an internet connection to help me.
If you want to see how well your agents performed, then send me the experiment id(top right on the screen) and I'll get the data for you.
Let me know how it went!

Controls:
WASD - move
mouse0/spacebar - fire weapon
O - mute music
P - mute everything
m - hide cursor if visible

Thanks! and I hope to hear how it went!
- Muhsin Kaymak

 

Here is a small video showing how this project started. In this video I kill agents that perform poorly until they learn how to reach the goal:

 

Webplayer

Hi Guys,
If you want to help me collect data, but hesitate to download the project, then you can also run the project from this webplayer:
http://sanjakproductions.com/ai/build/

However, I'm not sure how smooth it will run! So I still recommend the standalone version.

Thanks for the help!
- Muhsin Kaymak

 

Evolving Agents vs Human Player

In this experiment, the player stands still while shooting the Agents. Watch from 0:05 to see starting behaviours of the agents and 24:10 for the final behaviours of the agents. During the Experiment 320 agents were killed.

 

How exactly are you honing in on the wanted behavior? Is it using some sort of back propagation? or simply randomly weighted connections, and the strongest survive and you wiggle them?

 

The evolution theory is used.
This process happens each time an agent is killed by the player:

1. Two or one specie group is chosen among the best performing species.
2. Within these species; one or two parent agents is chosen among the best performing.
3. An offspring is evolved:
   1. If only one parent: the parent's genes are copied and mutation is applied.
   2. If two parents: crossover of these two parent happen. And a probability of applying mutation on the offspring.
4. The killed agent is replaced by the offspring and respawned into the world.

Choosing the best performing is done via a fitness calculation, that include these parameters:

• how long the agent was able to survive during encounter with the player.
• The shooting accuracy of the agent.
• The number of times the agent was able to successfully hit the player with a bullet.
• The ability of the agent succesfully reacting when encountering the player.

While choosing the parents, a roulette wheel system is used, which means the best performing agents have a higher probability of getting chosen, however, agents with lower fitness can also be chosen.

 

Looks like standard evolution algorithm. Is this enough for a master's thesis?

 

The difference is the inclusion of human interaction and that the evolution doesn't replace whole generations, but instead individual agents.

The fitness function in this project is more of a guidance that helps evolve better performing agents.
The real goal is using human interaction and how it can be used to eliminate agents that behave in an inefficient way towards the individual players.

So basically, I aim to have agents that behave according the individual players, instead of having agents that behave the same no matter the player playing the game.


I handed in my paper 15 days ago, so I do hope it's enough for a master thesis

 

Heya,

The reason I asked is, the technique you are using now is not very efficient at training any sort of AI agent.

Without some sort of back-propagation, or something more then "evolutionary jitter" to direct the evolution of the AI, you are literally leaving it up to random chance you will stumble across the behavior you are looking for and are more then likely just going to have a constant stream of stupid AI agents that are just awful. Even the ones that seem to "get it", will more then likely be some random chance happening then any actual sort of learning in the AI itself.

 

Backpropagation is very effective when the desired outcome of the training is known. However, if the desired results is not known beforehand (e.g. In this project, we don't know the best behaviour against the player), then something like reinforcement learning is necessary. The agents need to explore the available actions and determine the actions that result in higher fitness.

Evolutionary algorithm aims to start of with randomly generated agents that all perform different actions. As the agents perform their actions, they gain fitness, which is then used for determining, who survives and who dies.
By combining the best performing agents and generate offsprings, the goal of the algorithm is to gradually evolve agents that perform better than the previous ones.
Of course this will sometimes result in stupid agents in the population, but generally they should be better and better.

In this project however, the fitness doesn't directly determine who survives or who dies, but the player does. The fitness function only helps determining the agents that 'fought the best' and use them to generate offsprings.
Of course, sometimes the killed agent will be the one with the higher fitness. However, starting this project I had the idea, that the agents that is killed by the player must be the weaker ones against that particular player and using my implementation I would get better agents that were tailored for the player. This master thesis aimed to try to prove this point.

 

He is using NEAT and well known type of neuro evolution, was popularized into mainstream by this video: