https://www.linkedin.com/in/togelius

This interview around games and AI is with renowned researcher, Julian Togelius. Julian is Associate Professor of Computer Science and Engineering at New York University. He’s doing some fascinating, interesting research around artificial intelligence and video games. Remember how Google’s DeepMind learned how to play a video game on its own? That’s similar to Julian’s research but he’s advanced that even farther.

His research is around using artificial intelligence for the following:

-Adapt games in real-time to players ability or interest
-Make opponents more intelligent and believable
-An area called procedural content generation which is artificial intelligence tech that actually builds video games using a machine, fascinating stuff.

Here are some other questions Julian answers:

-Procedural content generation: How does it work? How do you train the algorithm? How does it learn?
-What machine learning algorithms are you especially interested in?
-Opponents research: How do you see your work (making opponents more intelligent) having applications outside of video games?

Transcript

Dave Kruse: Hey everyone. Welcome to another episode of Flyover Labs. This is Dave Kruse from Madison, Wisconsin and today we are lucky enough to have Julian Togelius with us. Julian is Associate Professor of Computer Science and Engineering at New York University and he is doing some super interesting research around artificial intelligence in video games. And so this research of course has implications within video games, but also beyond and some of this research is around using AI to adapt games in real time and making components more inelegant and in an area called the Procedural Content Generation, which is using the AI to essentially built video games using a machine.

So it’s pretty fascinating stuff, especially if you like video games like myself and machine learning. So this is like the best of both worlds. So that’s why I invited Julian on the show. So I’m really excited to have him today and we have a lot of talk about. So let’s get started.

Julian, thanks for coming on the show today.
Julian Togelius: Yes, hi, thanks for inviting me. Well, I’m happy to be here. Well I’m technically in New York right now, but here in the same place.
Dave Kruse: I wish I was in New York right now. I love New York. So yes, and maybe – before we get into your research, can you maybe just give a little background where you are from and how you got into doing video game research and that would be great.
Julian Togelius: Yes. So as you might hear from my accent, I’m not from around here. I’m from Sweden, grew up in Sweden, went and did my undergrad in Sweden. I decided that well, I finished high school. By the time I finished high school I hated maths and wanted nothing to do with it every again. So I decided to study philosophy and physiology at the University of London, Sweden, because I wanted to understand the mind and I did that and gradually I became more and more frustrated, because I’m not a man of very much patience and philosophy, I mean nothing really happens there. You read the text before the people and you sort of quarrel about the selection of words and so on. So I took more physiology courses as well and very dissatisfied with the matter.

So basically I mean you can only really scratch the surface, you can’t really get to the core mechanisms very easily. So I started talking some Computer Science courses as well. I had already sort of – for me like the other side was a bit of a self taught programmer, so it was fairly easily to get into it. Then I went to England, did a Masters in Evolutionary and Adaptive Systems. So basically the interplay between Theoretical Biology and Computer Science essentially and what I thought was that I still wanted to understand the mind, but I wanted to – I figured out I have to build the mind first in order to understand it.

So I mean Computer Science, I mean that was much more fun because things happened. You could build things and they sort of worked. So after that I went on to do PhD and start a company and so on, did a bit of social network and stuff before in the pre-Facebook era. I got a PhD at the University of Essex in England and I thought I was going to do Evolution and Robotics in using Evolution of Algorithms basically and win and evolution in the form of – in algorithmic form to design neural networks that will control robots. So I started a little bit down that path.
Dave Kruse: And what year was that?
Julian Togelius: They were still like basically…
Dave Kruse: What year was that?
Julian Togelius: This was, sometime – I started my PhD in 2004, so – and I thought that’s what I was going to do, but then I figured out that then it’s very slow growing in that as well, because you have to use robots and robots break down all the time and they need oil and you have to sort of screw driver some – I’m not – I’m still a very impatient person and not too very practical person when it comes to sort of physical devices. So I got a bit bored with that as well and then figured out that this thing was computer games, hey. So what I wanted to do was to bring in artificial inelegance, so I can understand the mind and I didn’t actually want to prove myself. I wanted evolution to do this for me. So I wanted to sign evolution around rhythms that could evolve neural networks, that could again, that would be intelligent, that can control robots.

And I figured out that lets do that, but robots are far too slow, so why not use computer game to start with. We can speed them up to like 1,000 times real time and the worst thing that could happen is that the game crashes, which his much less bad than if a robot sort of you know runs into a wall or something.

One of our friends and colleagues back then, he was evolving neural networks to fly miniature helicopters, and his room was like littered with pieces of miniature helicopters and I didn’t want to do that. So basically I went into that and they let you get out of – actually it’s true that you use games to test AI and its very much true and it’s like one of the main reasons I do what I do, but also figured out and I’ve done some really interesting problems in games, which I’m not the best paying in as well, but I got lots of other things that are more relevant for game development, game design and the game industry such as being able to create environments and levels or so on in games and model human players adapt games to players and that’s the sort of realization that led to a few papers that became sort of minor classics in this area and which really kicks out of my career properly.

Why I did work on the revolving racing tracks for particular players and so on, and similarly work on one of the all the rules for games. And I went to Lugano in Switzerland to do the PhD, or to do a post document after my PhD in machine learning. It was the same as machining learning was structured and I worked a lot with the current neural networks and so on over there. Then I moved back to Sweden and started working in Denmark, so I was commuting over a National Board everyday, which is something that when you tell this to Americans they are a bit like incredulous, What you’re commuting over a national boarder? Which is like, they imagine something like, Woo! You’re commuting from U.S. to Mexico or something, which was clearly not to be a very good idea, but its North Eastern Europe.

And I joined the group at the IT University of Copenhagen which was at a very, very strong game studies group. So I got that component and started sort of learning more about the academic study of games from various perspectives and have also games of social constructs and games as design. My research got closer to game design in various ways and now this is what I have here. I’m here in New York and I’m currently very much building up my resource group over here at NYU, working with the people at the game center, working with [inaudible] here at the Computer Science department and yeah, generally enjoying it a lot, so very much carrying on this ideal that I’m still interested in the mind. I’m still interested in what intelligence does, but I’m also very much interested in how can we make the game better through interesting AI techniques and how can we make AI better through interesting games to test AI, so yeah, that’s who I am now I think.
Dave Kruse: That’s great. And since we’ll talk about neural networks a fair amount, could you maybe give if you can a quick definition for folks if they are not familiar with what they are?
Julian Togelius: Yes, neural networks are – originally there were none to beat the very simplistic models of the brain, the brains of humans or of other animals, but that’s really sort of the mystifying way of explaining what they are. Actually they are just – they are just equations, a couple of equation systems modeling here. So what you have in your network is the mathematical object which has the number of nodes connected to weights and you put in numbers in one end of them and the numbers could mean – could be in like you know any kind of variable misstep and going to take a base or it could be the sensory readings from a robot or something like that.

And then on the other hand you get some modern – on the other hand of the network you get some smaller numbers out, which could be interpreted as some prediction about life through their expectancy or it could be interpreted as the direction in which they very much return or something like this. It all depends very much on the kind of auto that was used before and the trick here is to learn all the variables inside neural networks. These are called waves or connection waves or synaptic waves which we use the biological interpretation and are different ways such as – of training, so neural networks are very – I mean there used to be a couple in the 60’s, then they were very much unpopular for another like 15 years or so. In the mid-80’s they began for the rediscovery of the back propagation algorithm, which sets – which basically is the way of intercepting these waves, sort of training the neural network space using target data. So basically you can train in neural network based on lots of input and then lots of what kind of output you actually you wanted learn and then use the back propagating algorithm.

Then neural networks fell out of favor a bit in the 90’s, early 2000’s, because essentially its lots of people from a more mathematical background hated them, because they couldn’t understand them, because neural network had this property that, you know sort of black boxes. You can prove some things about them mathematically, and because they are in a sense mathematical objects, but all these mathematical proofs are sort of pointless or almost pointless. The theory doesn’t really help. It’s a bit of a black – so tuning this back box is a bit of a black art. You need to understand the neural network, certainly how to train them and yes, it’s sometimes more of an art than a science.

So other machine gun networks are starting to favor, but now since 2010 or something neural networks are back with a vengeance and super, super, the super hot thing and everybody is investing in them and what has happened is essentially fortunately that there are a couple of new tricks and a couple of newer networks. The details in networks are being invented and partially because we have some much better computing power and specifically because we can use graphic cards to do enough for the competition. But actually what we have is at the core of it, it’s the same due in the neural networks, we happen to hate, those are the fine principles and with a couple of more times versus them.

So these are a very, very versatile networks that we can use. We can use for training and you should use for learning also lots of different functions either from data or from feedbacks. Basically you can divide them up into the kind of networks that learn how to predict something based on lots of target data. We all have the inputs and you know what outputs you want or based on the feedback from the world – say you are training your robot to behave and the only thing you’re telling it is that that’s right, that’s bad you know, it’s get positive feedback for – for getting it from AXP and negative feedback for bumping into the table or something and eventually for them.
Dave Kruse: So what differentiates somebody who is in your mind really good at training neural networks versus not. You said it’s an art as much as a science and you know…
Julian Togelius: You need to have a standard. I mean, a part of it your practice makes perfect. You need to see how they are actually encouraging the behavior given lots of different circumstances. So you should open your networks, but it also really helps to be able to pull them up yourself and actually forget one.
Dave Kruse: Makes senses. All right so let’s talk about the…Oh! Go ahead.
Julian Togelius: No, no.
Dave Kruse: Yes, I was just going to say lets chat a little about your research and I’m kind of curious you are back through when you started doing a AI research around video games to now. What was that first project you worked on and curious how the research has developed and changed or improved since then?
Julian Togelius: Yes. So to start in one particular project, which was maybe not the best thing I worked in, but was important for me. So basically my advisor back then was interested in radio control car racing, the kind of peak, crappy radio controlled cars you can buy for $10 at the gas station and the other idea that we would teach and teach basically a system to race this care space with an overhead camera. But that will be very expenses basically in terms of time and also in terms of destroyed cost to basically train with the wheel base in cars.

So what I did was that I wrote up a very simple simulation, which was essentially a simple two dimensional racing games. I worked in Java and I wrote it – I mean I don’t know much about physics, so I just basically invented some physics and let’s say this was not the very good physics model, but hey, it sort of worked. So you could play it, got the players, and it was – I mean it wouldn’t awards, but it was a little bit fun at lease.

You played the game and it was sort of like getting around the tracks as soon as possible or as fast as possible. So then I started training, I mean neural networks to drive this car around the track and then what happens is that using evolutionary algorithms. So basically I had the neural network connected to a fewer sensors that in this car simulation measured how far it was to the wall at various points along the car, and also how far it was to other cars. And the outputs was essentially the steering and the accelerator where you go left, right, forward and how much are you accelerating or breaking.

And then I – then what happened was that we used evolutionary computation here. So you have evolution, you have evolution and trying – we had the population of say 100 neural networks and then you put each of these neural networks one at a time into the car as you to drive the car. Initially all the neural networks are random. So they don’t do anything useful at all. Most of them just likes sit there or maybe some of them just backed away or maybe one of them drives forward a little and then creases into a wall and it’s all crappy behavior really. But some of it is a little, little bit less bad than the other.

So you keep the ones that perform least bad, throw away the ones that are performed the worst and replace them with copies of those that performed, perform a little bit better and also you mutate them. Now sometimes you mix the neural networks operation, otherwise it’s called the crossover and is essentially analogue is having sex. So we let neural networks have sex with the shorter and produced offspring.
Dave Kruse: Nice, and there was – hurt that way.
Julian Togelius: So just like your – so basically it’s just like a biological evolution you get offspring which are very similar to both parents, but a mix of both plus are a little bit different because of random computing arrows. And then over time you do this many, many times like 100 generations and so on and what happens is that it becomes a search process.

You search for this space of possible neural networks that could drive these costs, so that you find those that – I mean they start driving the car forward and then they start and they start figuring out that actually if there is something in front of me maybe I should turn. And then they get even further, so the fitness function and we’ll watch them even further because of this and then they continue learning to handle more and more different situations. And then in the end you have really, really good mid-drivers and within the 100 generations, with a populations of 100, I would usually get neural networks to grow over this type and they play this game or drove this car better than I did.
Dave Kruse: Really. How long did that take? How long will that take to train?
Julian Togelius: Five minutes.
Dave Kruse: Really? Wow! Okay.
Julian Togelius: Yes, and this was like 12 years ago or 11 years ago. So basically it’s a lot fast now. So this is a very well established technique at this point, but back then it was clearly well established, but not that well established and I remember I got a – back in 2005 I got a best paperwork for this work and I was very happy and everything and then I went home and sort of worked more on it and tried to sort of make these neural networks be able to sort of progressively learn to have the more and more challenging tracks. So basically you start with training them to – teaching them to drive a simple track and then you probably more and more different racing tacks and in the end they can learn to navigate this really complex tracks and drive them really, really well and this is kind of tracks that are very hard for most humans and that these neural networks wouldn’t be able to drive at all if you just told them to do that from the beginning.

So this sequence, sort of a structural sequence became very, very important. It always makes you into like if you have different costs and they are all trying to compete with each other and you have one year network controlling the red car, you had it controlling a blue car and then their fitness is depending on not only themselves, but also on what the other car does. And then you can sort of evolve them to come up with all kinds of interesting behavior, ether behavior that sort of whether get it on peacefully share the road and both of them drive really fast or those that – if you basically were then not for getting as far as possible but only for getting as far as possible ahead of the other car, just to learn these sort of [inaudible] tactics or they will just basically push the other cars off the road and so on and it’s a carnage.

So really interesting stuff and desktop was really interesting also for, was usually for – I mean it could be usefully in game design in various way. But there is some point after working on this for a year or two or something, then I had to talk – I think it was one of these classical shout outs like why not do it the other way around. Why did you not turn it – things around. So instead of – I mean I now involving newer networks to drive cars on tracks. What if I instead evolve the tracks and which the new networks drive cars.

So basically – so if I have similar networks that drive something like humans, you can teach them to drive a bit like humans, by this back propagation algorithm. So you yourself drive the track a number of times and then you’re teaching your networks to drive essentially like you. And then you want to come up with a track. You can evolve the track the same way you evolve neural networks. You have lots of racing tracks.

Initially all of them are random when we made and then you basally, it makes more sense that you haven’t keep the ones that are least bad, keep the ones that – and throw the ones that are more and more bad and then more bad, really worse and you know you place them with slightly changed company software and that about tracks it and so on. And you do that again and again and again and again and again and in the end you have some really nice racing tracks. That is if you know what it means for a racking track to be good.

And now what we are approaching the general problem of actually automatically telling and saying what good game design, because what you want – if you are playing a racing game, what do you want really, what do you want for the racing track. And this is not straightforward at all. We started thinking that well maybe you want variety, you want some products where you have to, where you can drive fast, some cars where you can, where you have tricky curves, you have to navigate, maybe you want racing tracks which differentiate as much as possible between different drivers etcetera, etcetera.

So we are trying to do a couple of these things and this work became pretty influential and then I start thinking, well you can do this for most of other things. I mean yes, basically that was the end of that project, but then that sort of took the same ID into all market kind of aspects of games. So not only sort of racing tracks, but level apart from games like Super Mario Bros or Maps for Start Craft or games rooms for very simple games and in themselves simple packman like games and so on and I don’t know. This is a peculiar topic where you can sort of wish it can be all cheap digging, because there are so many more things you can apply in this basic interest.
Dave Kruse: Yes definitely.
Julian Togelius: That was so much, yes I was going to talking a senior project, but that sort of derailed a bit.
Dave Kruse: Which project?
Julian Togelius: No, I was going – you asked me to talk about a senior project…
Dave Kruse: Yes, that’s right. No, that’s great. I mean that was helpful enough about your explanation and the first, the project – my first question was I mean what were the inputs, outputs, but you nicely explained all of that and how it all works. So that was quite helpful. So you have a few areas of focus for your research, but it seems like they kind of overlap quite a bit, whether its…
Julian Togelius: Yes.
Dave Kruse: And so when you are working on a new project, do you kind of – do they all kind of mesh together or do you say, hey, I’m just work on procedural content generation and try to figure out how to create new levels or new games. How do you come out of your research?
Julian Togelius: So, I had good projects come from this. To your second question, I am generally motivated by – I mean when I see something, when I see something that I don’t really understand, well when I see something I don’t understand how to do it. So recently I was teaching a course on AI games and we are talking about different ways of playing different games and so for example recently a method called Monticarbit [ph] research has become very, very popular and we use it in different applications and its very useful for playing games we don’t know the rules and the manners like we’ve generally been playing. Those are very useful for games such as Go.

So as you might know, I knew Jeff for long time was the – he was sort of the pinnacle of – he was thought of as the pinnacle of inelegance like that basically if you can play chess you much be inelegant. Then in 1997 IBM comes around and takes the city grand master Aki in chess from [inaudible]. It basically shows that the computer can play chess that a normal human can and the algorithm to do this is very, very simply. But then we have a classification board of Go where – which really maxed the algorithm that basically could play chess better than a human. It couldn’t really deal with that because you have too many – the ground seemed like it was too high angled. We had too many possible moves.

At any given point you can make around 30 moves in chess where as you can make around 300 moves in Go. And the algorithm that eventually sort of conquered Go was done in Monticarbit research and Medicine. It is a fantastic research algorithm. Basically it looks at all the possible future moves and all the possible consequences of future moves, but of course you can’t look at all of them, so we use the statistic to decide the results and must tactically decided where to look, which moves to play out and which moves to look at the consequences of.

So I was teaching about that in a course and then there was, one of my students wanted to do a project were he wanted to apply this algorithm to a strategy game Hero Academy. And Hero Academy has this nice topic that you’re not moving only one unit, you are moving many possible units. This is the truth of many different strategy games. So we looked at how we could use this.

As I had done with the student then we talked about okay, multi talent research has been very useful for so many games and let’s look at how it would work for this game. And it turned out that this was pretty hopeless, because – whereas chess had a ground factor of 30 different moves you can make every time around and Go has a ground factor of round 300. This sort of innocuous looking strategy game, that didn’t seem that hard for humans. If you talented all the possible things you could together, it became like many millions of possible moves. So it was like Wow! This is crazy you know and if we just took this multi talent research or algorithm and applied it, it just wouldn’t work at all. I mean it wouldn’t get anywhere at all.

So we started looking at a couple of different modifications onto this algorithm, but we also thought of why don’t we just think of this problem from another perspective, instead of looking at you know which move and which move does each unit make and then select them one at a time, why not look at you know if you take all the possible moves that all the units can make as sort of one things, one object, whether the one which is we can have like you know many millions of, then you can search the space of such subject. So you can search the space of possible times and then we though really, maybe be we can use evolution here. So we tried that and it worked like a charm. It was like amazing how it sort of you know beat all the other solutions and by huge margins.

So we recently go a paper published in that, a paper in a conference a few months ago and yes we were right about how to make sense. We are definitely going to keep exploring that. Currently we are looking at it that it can be used for StarCraft or something, or as it seems like maybe it can. StarCraft is this real time strategy game which is famous for being extremely hard for computers to play and it’s pretty hard for humans as well, but there is another sort of competitive high level play, and so that’s one particular example.

Just looking at, maybe one of those you know have to do something and then trying to look at the methods we have, look at the perspectives we have and see sort of what new thing we can, what we can pull out of the hat and adapt to sort of make this work. But it’s also very clear that very often I mean the solution to one problem sort of you know comes with a number of new interesting research questions. So like the solution to how you – how do you drive a car in a racing track, how do you learn to do that sort of suggests that a couple of other interesting questions such as how do you all fell about racing track.

So I think, I mean my job is sort of some as the leader of a research lab or our research group, it is essentially to keep this overall picture in my head and connect everything to everything else. So if I have a solution to this problem, well maybe this has something, answers about this other problem for example.
Dave Kruse: Interesting and what the – I’m just curious, with the evolution algorithm, how is that different in a typical Neural Network?
Julian Togelius: Yes so these are interesting different things, because basically, I mean your network is a computational structure that basically maps inputs to outputs. And evolution in this sort – so in an evolution algorithm in this context is more like a way of training it. You can train Neural Network in many different ways.

So if you have, so basically for example, if you have target data, if you know what you want the outburst to be for a lot of different examples, then you want to use then versus that propagation. So in most deep learning applications, you are training something like for example here is like you know a million people. The data base of everybody who lives in some city. And we have lots of inputs which is telling them where some interesting things are and about them, the age, the height and gender, whatever and then we want to predict whether they are smokers or not and then we have the target data, that’s whether you smoke or not – whether you some at all or you don’t smoke.

So you are training all of these and you are train the Neural Network using that propagation on these examples. You input the example and then you say whether they smoker or he doesn’t smoke and we correct that. But when you feed back the correction into Neural Network and that changes Neural Network a little bit, so it give a little bit much prediction and you keep doing that all the time. This is called supervised learning. Supervised means that there is something that supervises the net and that tells us what’s right or wrong. And also what the right answer should be.

Then there is something such as the enforcement learning, setting where you don’t know what the right answer should be, instead you just, you just know when you are doing well and when you are doing badly. And in such cases you can use something which is called temporal difference learning and/or Q learning which is quite popular these days.

So just sort of feedback and so basically give is punishment when it doesn’t do that, we warn them when it doesn’t be good, but you can also use Evolutionary Computation and Evolutionary Computation sort of works at a higher level, because its try to search the space of the whole network. It doesn’t do sort of – it doesn’t sort of look at what, what feedback it got on an individual sort of prediction. It looks at the whole network and puts it out, aired and left to sort of do its job and then basically keeps the one set and tossed away the bad ones and so on. And sometimes this can be a very inefficient process, but a beautiful thing about it is it’s an extremely generally process that works for almost everything. So evolution can be used for training new networks and for lots of different things as well.
Dave Kruse: Interesting, okay. That’s helpful. And I read some place you mentioned on your opponents research where you know you use AI to essentially increase or decrease let’s say the level of opponent that he might be playing on a video game and you said that this research had applications outside the video games, I think I read. What did you have in mind there?
Julian Togelius: Yes, so basically – I mean when you look at game research there is some people that think that well, game research, why isn’t this guy doing anything useful with his life and you know, why is he wasting his time and someone’s money you know and this is ridiculous. But I mean first of all, even if game research has no new set of games, I still think it would work for one, because games is likes an important part of the culture; it’s like becoming the most important part of culture I think. I mean people don’t worry. I don’t think I’ve really been watching much of TV and the next generation is not going to watch TV at all, just the game stuff.

So it can, because there is just so much money, because it’s so much more attractive. What else can you say? I mean games are also being found to have, all this learning effect. So games are increasingly being used as medicational sometimes, you teach things in the form of games, because games are fun to a large extent, because we are learning as we are playing them. So games are masters of teaching you when you play them, how to play them, because games that’s real is no fun at all. A game that’s impossible and we can’t make progress is no fun at all. And those games which most – I mean a very large portion of why we think games are funnier because we are gradually getting better at them.

So they are sort of ultimate learning masters of the game or ultimate teaching masters of the game and this aspect of games is very much being used in schools and work place and so on to teach all kinds of other content, which is not usually thought of as being inside a game. So teaching mathematics and teaching history and so on.

So there has to be sort of some serious games that have very lean applications. But then we look at it outside of games as well. I mean you have – every game is a model of something. So if you are playing a Flight Simulator, it’s a matter of time. If you are driving games it’s a model of driving, if you are playing Civilization, which is a big strategy game, then it’s basically a model of managing the country or something. And if the method could be used in this game, it could also to some extent be used in the sort of natural process of this modeling.

So if your – for methods for driving a car for example, it could of course be usually in terms – so I’m talking cars and also actually I think now I know people who are training neural networks to drive costs, to driving simulators, which could then be used in actually self driving costs that could be driving – you know that will be driving on our roads within a couple of years. And if you look at other games like strategy games are obvious to some extent based on military conflict and if you want to come up with a good tactic, a good strategy automatically, well then that is going to be vey usual for you as well.

And I could go on, you know.
Dave Kruse: I bet.
Julian Togelius: I mean you have the elevation, elevation to sports, elevations to all kinds of problem solving and yes, what have you.
Dave Kruse: Yes, well I mean you made the point at the beginning where you went from – and it was smart, from robotics to video games so you could just test stuff so quickly. And there’s very few environments it seems like where that’s the case. You might have like this large data set, but then that’s the data. Here you can like generate new levels, you can keep testing and yes, it’s a really interesting space to – and I haven’t thought about it much using video games to improve neural networks and AI, but it makes sense.
Julian Togelius: We recently launched I mean – I’ve been evolved in a number of – setting up a number of sort of game based AI competitions. So we build an AI competition from a game and then people submit to this AI controls to play the game. So we build one racing game, and we build one basis for Mario Bros or several versions of the one basis of Mario Bros. Recently we have saw that people tend to specially too much on a particular game problem, if there is a problem, because we get much more next generic solutions, because inelegance is not being able to solve one problem well. Intelligence is being able to adapt to any problem that you might see and that’s why basically chess playing computers such Deep Blue that own the grand master title from the best human, but can’t do anything else than playing chess. It’s not a very intelligent program, because it can’t do anything else than playing chess.

So recently we set up this thing we called General Video Game AI competition or General Video Game Playing Competition where people submit their best general – in playing AIs and they can train them on – like currently we have 60 different games and training sets and then we have tested a number of new games, which we don’t know before. So basically doesn’t know what games they are going to play when they developed AI.

The AI doesn’t even know what game it is playing until its [inaudible] sort of you know, here you go. And then we are testing them on the capability of playing new games that I have never seen, which is pretty cool. And I think this is becoming more and more, more about becoming the future, and it is also becoming very popular, this compilation, so people are developing more and more agents and sort of sophisticated solutions for this. It’s really sort of driving the research into a risk of some kind of AI mythologies. So yes, I mean we’ve got like training – training the algorithms not on a single problem, but on a whole number of different problems. So it is flexible and can adapt with anything that you know life gives it.
Dave Kruse: Interesting. So with the procedural content generation, how – I think I read that you want to make some of these algorithms more creative than humans or maybe a different creative. And how – so how does that work? Can you explain if you have like a PCG project going on, like how do you start training the neural network to design something from scratch.
Julian Togelius: It’s not always a neural networks. I mean you don’t always have to have a neural network. So in this case – but evolution its very commonly used in this case. For example, when it came to StarCraft games, we used evolution and computation. We have a number of very simple stock-up labs and then we as a business function to tell how good they are, and we tested how players have been doing in various ways. So we tested how much we have, whether you can go get from one base to another, whether our show funds are underway, whether our resource is on and [inaudible] to basically those evolutionary methods. But there are lots of other methods as well. You can use constraint satisfaction, which is the mathematical algorithm that tries to – it also does a kind of search.

Most of these algorithms does some kind of search for game content. But basically constrain satisfaction makes that searches for, if you slap a number of constrains that a good company needs to satisfy and then the algorithm starts to compensate for good levels of maps or something that satisfy all these constrains and those methods are based on grammar expansion, where you define a grammar that can be expand. Basically like English grammar can be expanded in general sentences. You can define a grammar that can be expanded into game levels or vegetation to take the game to another level or personalities or something like that. So there is lots of different methods available here. I’m so much partial to using evolutionary computation but, yes.
Dave Kruse: Okay, interesting. And what’s the input, for when you are designing a game. What are you feeding the area?
Julian Togelius: Yes so basically the big problem when using evolutionary computation to the assigned level is to decide – there’s two problems. One is how you are going to represent them. Basically if you are going to – I’m taking game level as an example, but it could be lots of different things. So think – say you’re at Super Mario first level. One is like, how you are going to hold this level or represent this level in numbers that can be manipulated by the evolutionary algorithm. The other is the evolution or fitness function. How do you tell what’s a good Super Mario first level?

So we’ve don’t research for example when we looked a couple of very common design features from Super Mario, and this was my first level. So this patterns we call them, which we identify by hand and then we – and then we search the space. So we put them into the fitness function. The fitness function is how many different design patterns does this level express. And also, is it playable. Can you get from the end to the end, from the beginning to the end without being impossible gaps you can’t just over or something and then you search space for levels that express these different patterns.

So there is no really input as such, because basically you have absolutely random levels at the beginning. The real trick is finding out good representation that can be searched and to find a good fitness function. That’s the really, really complicated part. Because finding a good fitness function for some kind of creative domain, it’s like saying that you automatically can recognize what’s a good creative products and what’s not. It’s like you know, say you wanted to good fitness function for paintings. Well that would be like being able to tell what’s good art from bad art and no one can do that really. So it’s really complicated and I mean to tell me good Super Mario Bros first level from bad Super Mario first level is easier than them saying what is actually – Pablo is actually a good painter or not, but it’s very, very far from trivial.
Dave Kruse: Interesting.
Julian Togelius: So we are looking into actually using – learning that from data, looking at lots of existing Super Mario first levels for example and automatically finding and training in this case that’s to recognize what we determine, what we have set the good ones apart from the bad ones and then using this function to train basically.
Dave Kruse: So…
Julian Togelius: And this is also what I’m talking about, where I’m talking about many computers being more creative than humans. Because humans are not very creative, and you tell the humans to be very creative and human will usually, basically copy something some other human did and they are very convention kind. They just do things that – they do things according to all classic conversions that you might not even know about. Like of course you draw straight lines, and a square always has four corners and things like this. Where there are search algorithms that is an evolutionary algorithm and doesn’t give a shit essentially.
Dave Kruse: No, no.
Julian Togelius: It sort of. It’s just test various modifications and tries to now think of it as stunt like and like Oh! Interesting, let’s go this way.
Dave Kruse: And one of my favorite parts I read on the GO with Google, being the GO champion is that some GO experts looked at all the moves and the Google Go player was performance moves that they had never seen before or they even think about. And so to your point, it’s just, it creates something entirely new essentially that humans haven’t thought about.
Julian Togelius: Oh! Yeah its true, because it doesn’t think the same way.
Dave Kruse: No, no.
Julian Togelius: So it doesn’t really – it doesn’t follow the conventions about whether what a good GO move should be.
Dave Kruse: No. I know we are coming to the end here, but I was just curious about – what the Super Mario Bros as an example, so like would you essentially map out the levels, so like a pipe would be or a tune would be a certain number at a certain location on coordinates.
Julian Togelius: Sort of. Yes. So one common interpretation that is used which basically is that we divided them into slices and we basically look at like things that are sort of one time divided, one time is like half a product for the world of Super Mario and stuff and then we sort of sliced up the levels and then we figured out that lot of strategies that we used in other time are actually we only need like 25 or so to cover like 95% of the original Super Mario Bros games and items. And one more thing sort of just we understand Super Mario Bros is the sequence of these slices of this Super Mario Bros level to typically their representation would be like you know wrapped positions. So basically like just list of numbers like 0, 12, 15, 00, 1, 18, 2, 2 etcetera, etcetera and now we can search this space of such lists of numbers.
Dave Kruse: Interesting, interesting. Okay. Makes senses and – go ahead.
Julian Togelius: No, no go ahead.
Dave Kruse: I was just going to say, so this is probably the last question since we are running out of time here and definitely appreciate your time chatting and I was curious what type of – maybe some of your current projects if you have. One that you are want to share. I’m sure your interested in all of them but…
Julian Togelius: Well true.
Dave Kruse: Is there one especially.
Julian Togelius: I’m super interested in – right now I’m very interested in learning the sings and style for particular levels on a pretty granular level. So taking the StarCraft level for example and trying to predict features from one level. I mean predict some aspects of the level form just [inaudible]. So basically you have – you just feed in the pixel of the StarCraft level and trying to figure out how many times does this StarCraft allowed to take or does it – is it easier for as a talent or produce to play this. These are different sort of actions or races in the start-up game or will this game be won by particular strategies or there is some other strategies that are going to be more effective in just training this gives some more data. But also like you know, looking at like can you take just a part of this level and predict the rest of the level from it, so instead of a complete the level, if you only have a slice of it.

This is useful. Both of these is very useful, because I mean I do think that, I’m working on – if I talk to academics, I tell them that I’m going to replace all game disorders with algorithms, and everybody is happy. And so basically I mean all my family are artist, so I wouldn’t be afraid of making them unemployed if it wasn’t that doctors don’t make any money, anyway.

Now if I talk to game designers, I’m not telling them I’m going to try to set up sort of al the creative stuff for them. I am telling them that I am making software which will help them with their work, because I mean when you. The way we put AI, against you I’m simply sort of [inaudible] And then you ask like who is going to win the AI or the human. And you are sort of doing us all a service in ourselves because actually what’s going to win is the union of AI and human. So basically than humans augmented by AI that’s tough as much as possible, all these things that AI does better than humans.

So for example for designing games, I’m thinking of coming up with systems that can sort of becomes one with you and asks you design a game. It helps you in all kinds of ways. It sort of created a leverage or that does your bathing and helps you in all kinds of ways. So you are designing a game level and the system helps with your suggestions. Maybe you should put the base over here or well, I’m not sure you should put this mountain over here, because that basically makes the base survival. Strategy is much smaller or maybe currently this level, reintroduce this event with just players that play this particular fraction. Maybe you should add like on observation tower over here and or something like this. And constantly gives you feedback, just sayings to us, check the level for your various ways. So it becomes like – so it becomes one with your creative expression, your creative through this AI system in the sense.
Dave Kruse: And that has implications. I mean in any different field, any type of design creative, whether that’s software or like you said art or lots of spaces. Interesting, and so…
Julian Togelius: I mean our tools define our creative expression all the time. So basically the new generation of tools who just have AI in them, and the intelligent than themselves. And we are going to be, I mean functionally we human creators are going to be sort of be joined with this AI and power tools.
Dave Kruse: Well, that sounds exciting. I’m ready to be AI empowered.
Julian Togelius: Yes.
Dave Kruse: I like. I’m going to check out the – follow your research closely. So when you start publishing into this, more in this space that will be really interesting to see. Well, I think we have unfortunately come to the end of the interview. But definitely I really appreciate your time surely and this has been fascinating for me and…
Julian Togelius: Thank you. I always appreciate it. Thank you.
Dave Kruse: Definitely and yes, I’ll keep a close eye and hopefully the audience will tune into what you’re doing. And thank everyone for listening to another episode of Flyover Labs and definitely appreciate it. And bye to everyone and thanks again Julian. Bye.
Julian Togelius: Bye.