https://www.linkedin.com/in/jeff-palmucci-08042b16

This great interview with Jeff Palmucci gives us a glimpse into how TripAdvisor is becoming more intelligent. Jeff is currently the Director, Machine Intelligence at TripAdvisor.

Jeff has a super interesting background. He has started several companies including Optimax Systems, which was acquired by i2 Technologies (which has since merged with JDA). Jeff got his bachelors degree from MIT. He has also been an angel investor for over 18 years. And has run a programmatic hedge fund.

I invited Jeff on the show to hear more about his background, what he’s excited about now and what he’s doing at TripAdvisor.

Other things we talk about:

-What did Jeff learn from running a hedge fund?
-What types of machine learning projects is he working on at TripAdvisor?
-How does Jeff figure out what to focus on?
-Where does Jeff want to take machine intelligence at TripAdvisor?
-How does Jeff understand what’s a romantic restaurant, and recommend restaurants you might like?

Transcript

David Kruse: Hey everyone. Welcome to another episode of Flyover Labs and today we are lucky enough to have Jeff Palmucci with us. And Jeff has a super interesting background. To be honest, it’s hard to know what to include in this short intro here, but Jeff is currently the Director of Machine Intelligence at TripAdvisor, which we can talk more about of course and Jeff has also started several companies including Optimax Systems, which was acquired by i2 Technologies. And Jeff got his Bachelors from MIT and he’s been an Angel investor for over 18 years and has run a hedge fund. So he’s got lots of things we can talk about. So I invited Jeff on the show to hear more about this background, to hear what he’s excited about now and what he does at TripAdvisor. So Jeff, thanks for coming on the show today.

Jeff Palmucci: Great, thanks for having me.

David Kruse: So let’s – as I said, you’ve done a lot of interesting things in your background and we won’t be able to talk about all of them in detail, but maybe could you give us a little bit of an overview on your background so everyone can get to know you a little bit better.

Jeff Palmucci: Sure, sure. My background does go back quite a bit. I guess I started programming professionally at the age of 11. My dad actually bought an awesome IBM 5100 mini computer that he put down in his officer, using accountancy. Then I thought myself how to program on it and at the age of 11 he gave me a job of running accountant software. So yeah, it was – I can’t stress the importance of you know if you’re kid is interesting in this kind of stiff get him involved early, because you know kids suck up to this you know this kind of information really, really quickly and that really benefited from having that opportunity. I had a bunch of Geeky jobs through high school with a computer background and went to MIT, spent a lot of time in various labs, I tried to start at the labs that I worked at for a while as a government contractor and the MICR Official Intelligence Laboratory; that’s where I started my background in AI and machine learning. After graduation I went to Bolt, Beranek and Newman, another government lab where we did a lot of research to a whole bunch of different areas, machine vision, natural language speech, that kind of stuff. Did a couple of startups, sold two of them, started up more of them, and then not everyone goes well. I came back to MIT with now a C-Cell and did a bunch of research in question answering systems there and from there on to TripAdvisor and like I run the Machine Learning Group here and we do a whole bunch of machine learning projects across the various products areas in the company.

David Kruse: All right. So in your past, I am sure you working on many projects and I would love to hear about all of them, but probably we don’t have time for that. So do you have one or two, and it could be part of company you stated too, one of two that were kind of most memorable or most meaningful with something that you learned a lot or yeah.

Jeff Palmucci: Yes. So, I’ve been pretty lucky and that a lot of the jobs that I had you know have been with the really smart people and you know I was able to learn a lot in a lot different areas. Like BBN, I was at BBN Labs which was my job right out of school. We did government research and did research into machine vision and scheduling and natural language and you know it was a really, really interesting place to work and that everybody that was very, very smart and you know that’s where we spun our first company out of. So the first company Optimax that I was the Founder of, we did a production scheduling software, we developed some technology in BBN and for various reasons it wasn’t really working, marketing from inside the company. So we spun it out and they get a chunk of the company and you know we were able to make a commercial success out of it. Optimax was an interesting company, because it was more operations research than machine learning and that we were working on scheduling large factories, right. So the first job that we ever had was John Deer planters. And the things about planters, you know the things that you drag behind the tractors that put the seeds in the ground is they are all different. For example, every single order is just a little bit different, so it really matters how you schedule these things going down the assembly line. And so we have this you know algorithm that we developed at BBN and took it out and then installed there as a scheduling system for the first time and just with the scheduling system in place the throughput of the plant went up 15%, so just the software right. And after the software was also very good as calling out constrain that they had, if there was constrains in the plant that they had. So after we fixed some of those, they went up 45%. So yeah, no that’s was crazy.

David Kruse: Guess you got the contract. Yes I guess that…

Jeff Palmucci: Yes, no we got the contract and then we rolled through the rest of John Deer, right. So we had talks up and we rode through the case and then we wrote to Volvo and GM and you know after four years we sold the company to i2. It was more of the manufacturing in the planning spaces as opposed to the scheduling space and it was actually a pretty nice acquisition all around, so yeah.

David Kruse: And I’m curious well – and we don’t talk the whole time about this, but what were the inputs and how did you just, how did you get that such huge increase just from software?

Jeff Palmucci: Well see, if you think about how things rolled down in the assembly line and how – especially when they are all different, there is going to be different subassembly lines that are stressed more or less based on the different products you go through. And so you can by just arranging the products as they go down the line, you can try to load balance between all those sub assembly lines and keep things running very smoothly. So if you think about, let’s say that I got like a 12 year old planter and another 12 year old planters, big ones going down the line, it’s going to really stress this one subassembly line a lot and they are going to get backed up right. So because there is no inventory between these stages, it just holds up the whole line and causes a problem. And then this particular plant was a really – it was revolutionary for them and that they just rolled the planters off the assembly line and right into trucks you know at the lot. There was no inventory at the end. So that made the discussion of the problem even more interesting that you know there were all these constrains flying around, there were all these things that you know humans schedules would have to keep track of and humans just aren’t really good at keeping track of all these little details and computers are very, very good at keeping track of them. And we were just able to you know – we were just able to help those guys out and make the plant run a whole lot smoother, so yeah.

David Kruse: Interesting. So how did you get the inputs? Did you use their existing software or did you like…

Jeff Palmucci: So it wasn’t really a machine learning problem in this particular one; it was more scheduling. So the inputs were the orders that were coming down the line, how they stress the various subassembly lines and you know different constrains on how things should flow down. This one should be close to this other one, you know because the system server line might get overloaded and you know it was very interesting because the workers would get around and negotiate over the constrains that would go into the systems and it was really interesting team work and that you know they could put constrains in the systems and build the schedule and you could see how things would kick out of the schedule because people were bold and they would iterate and say okay, well I guess we could do this if you went with these changes. And you know it was literally, it was sort of successful prototype, yeah.

David Kruse: Interesting. And what was the other company. I didn’t mention that one that you started and eventually sold?

Jeff Palmucci: That was Percipo Capital. So yeah, that was a hedge fund. So that – so after we solid Optimax, I was there for I guess four more years at i2 and then me and my business partner started doing research into machine learning and we found some algorithms that did very well in very high dimensional space. And you know, so we kind of like had a hammer in search of a problem and we looked around the worst scenario, hooked up these finance people and built a hedge fund around the algorithm. You know hedge funds are an interesting and challenging place to do machine learning, because you know the difference between making a lot of money and losing a lot of money can be like you know 51% – 49%. You don’t have to be right a lot; if you are just right, just a little bit more than average, then you could just make a ton of money. So therefore, you know everybody is trying to take their shares out of the market, right. So the signals get you know more and more challenging as more and more people look at it. And so a lot of what I’ve learned in hedge funds space is how to counter the kind of data snooping that’s kind of prominent in the area right. So you know look at a set of data a number of times and you are going to find signals whether you treat it or not right and you know that’s especially true, that’s especially true in finance where all sorts of people have all sorts of different theories and they throw the stuff up against the wall until something sticks and go with it. And if you don’t have good back testing procedure you can really sue yourself into a different kind of a signal where you really don’t, yeah so…

David Kruse: Interesting, was it.

Jeff Palmucci: So that’s very different from where I am now with TripAdvisor, right, because at TripAdvisor we don’t – we have so much data it’s just crazy. You know if we want to run a test, then we wait a week and we are going to test it, so.

David Kruse: So did you – was it stressful, did you enjoy running a hedge fund or?

Jeff Palmucci: So it was stressful in that area, okay, in that time because this was when the great recession was starting. So the reason why we pivoted was simply because the market tanked and no new fund could raise – and we had some money under management and no new fund could really raise any money and even though we are showing some reasonable results, we just couldn’t rely on that kind of management. So we actually pivoted into internet marketing right and Percipo Capital is now a Drug A, so they are still a very successful company.

David Kruse: Oh! Interesting, okay. So you pivoted the job okay. So what – yeah and I suppose a hedge fund is kind of the ultimate in competition in the sense that everyone has their algorithms out there, all kind of best east other.

Jeff Palmucci: Yeah and you notice as you look at the stuff for any reasonable amount of time that the market changes you know. There is regimes where the market behaves one way and then we were doing mostly commodities trade, so this is the commodities market that I’m talking about. And then it flips a switch right. So we are running this one fund and running for about six to seven months and it was doing okay and we notices that some of our stats that we used to watch, to – some of the stats that we calculated were going into like two or three single territory. And like oh! Something is wrong here. So we yanked the fund and fortunately we did that because you know after the signals started to go haywire the market, the market tanked like 10% and yeah.

David Kruse: Oh! My goodness.

Jeff Palmucci: Yeah, so sometimes you win by not planning the game, you know I guess that game was one of those.

David Kruse: Oh yeah, as long as the smartest people know when to take the money off the table and what – so how would you come up with an idea and then test those ideas and then how would you know that you did back test it properly.

Jeff Palmucci: Well I mean, okay, well back testing, you have to be very careful of back testing. So you need to keep a set of data aside and this is what the real raw machine drops as much as it plans. It’s just a lot harder in plants because the signals is so much weaker. You need to keep a set aside that you do not touch until you are ready to take the product to market. And then you love the lunch and then for every time you run against that data you got to realize that there is multiple players going on there. And if you just take you know, do draw an emotion like just flip a coin you know and say okay, figure some time when you walk this day and figure some time I’m going to go down this day and you know do that three times, you are going to have a fund that looks pretty good you know.

David Kruse: Yes.

Jeff Palmucci: So it’s very you know, your funds are you know – machine learning models in Missouri are very sensitive to those kind of multiple abrasions and so you have to be very careful with it.

David Kruse: And how do you come up with new ideas, like you know will you just sit around like hey, we should see if this commodity correlates with this index or how do you – I won’t even know where to start.

Jeff Palmucci: There – so there were a bunch of training signals that we used for the commodity funds and I guess you can play right now because we took it out of the market. So if you have a bunch of traders that are following a trading signal and the trading signals all pointing you to do something like buy, for God sakes don’t buy right.

David Kruse: Right.

Jeff Palmucci: So you know the technology that we are using was not really the neural network; it was a very – it was a rule induction system. So you could actually see what the model is reaching. So there is certain amount of, does this make sense that went into it and you can’t really do deep learning right now. But there is a lot of other machine learning techniques where you can take that kind of version, so be careful about your back testing. And you know all machine learning is – even the deep learning where you go inside that deep neural networked. Nobody really knows what’s going on and you are putting together a structure that you think can be successful right. So you know any of this doesn’t allow you to get – you got to know the Derain good enough that you can hypothesize that this particular model structure would work and then test it.

David Kruse: Got you, okay. All right well let’s talk a little bit about TripAdvisor, your current role. Can you tell us kind of, well what your role entails now and your focus areas are?

Jeff Palmucci: Sure. So the machine learning group here is pretty interesting and that we kind of consult to other group, to other product group in the company. So we do a lot of work in a lot of our product areas. We’ve done work with a review for arbitration, a lot of interesting stuff there. We do work on organizing our site search results. So ranking the results, we got a lot of success in that recently. Core understanding, so you know if somebody types in you know Chicago Pizza, they are looking for Pizza places in Chicago or they are looking for a restaurant named Chicago Pizza; doing that kind of parsing. We do lot of a classifications. So you know we have over 300 million reviews right now and that’s a huge amount of information about the different locations that we have in our database. So when you go to TripAdvisor and ask to show all the romantic restaurants or all the Italian restaurants or all the business hotels or all the family friendly hotels, those are actually all machine learning modals underneath there that we train based on votes that we get from our users and the underlying review text. So there is lots, there is about 300 different models right and making all sorts of decisions about our locations that’s all driven on machine learning.

David Kruse: Interesting. And before we get too far – I was going to say, before we get too far, do you mind just giving a brief overview on TripAdvisor. Most people probably know it, but in case they don’t?

Jeff Palmucci: Yes, so TripAdvisor is a number one travel site on the web. You come to us when you are looking to plan a trap and looking for interesting places to go. We’ll look out for the best and you could compare prices and book on TripAdvisor. We have millions of restaurants online, so if you are – once you are on your trip or even if you are home and want to find a good place to eat, we have names of the restaurants with tons of reviews under those. And also we have a lot of things to do. Once you are on your trip, what are going to do when you are there, there is a lot of information on that stuff. You know the machine learning in this particular domain is a really interesting problem, because you are trying to organize the informant and let say for somebody that’s going somewhere, who doesn’t really know too many details about where they are going to. You want to help them discover their prefect trip, right. So you know a lot of the work that we’ve done in machine learning is trying to organize stuff on this site, so that we can surface interesting things to our users, right. So you can – we’ll cluster things in interesting ways, so that if you are looking for a hotel in New York, are you interested in finding a hotel or are you interested in seeing Broadway shows. We would try to use a bunch of machine learning in order to try and surface these relationships, so that when people are taking their trip not only can they judge the quality of the particular location, but they can you now discover interesting things to do once they are into it you know.

David Kruse: And how do you do that? Dou you – you have a ton of people like you said that help on a large amount of data. So like Dave, you know maybe I like going, I love going to Manhattan staying at the Marriott and I like going to these certain restaurants, so then do you take that information, like oh well then you know another…

Jeff Palmucci: Yeah, so let’s say that you are looking for a restaurant, you know a lot of machine learning goes into what type of food are you interesting in, what’s your price range. You know all that stuff is very high information to come by in a catalog, right. But we have a – we have millions and millions of awesome users that give us this information and TripAdvisor through machine learning and through just reassertion helps organize this in to you know, into actionable plans so that you can go and find a place that you want. But we are working on a lot of – we are working on a very interesting problem right now, which is once you are in New York City, you know sure you found the restaurant that you want, you found the hotel you want to stay in, but what are you going to do when you in New York City. You know I’m in town for the weekend; how can I find some interesting things to do, to really see the town. And this is something that we think machine learning is very applicable to, because there is just so many different places in the world to go to and we have all this, we have all this information that people are talking about places and using machine learning we can surface interesting things that people are talking about so you can find them.

David Kruse: So where would you – so let’s say TripAdvisor just brought you into the company. They don’t have anything, no machine learning yet and they are like hey, we wanted to you know get better at discovery you know. Tell people what restaurants to visit and you know match their interests with what restaurant then. And so how, how would you kind of start a project like that and get it up and going and bringing that data together and yeah, can you kind of describe how that would…

Jeff Palmucci: Yeah. So I mean there is a couple of different programs that help, that help do that right. So we talked about the classification examples, where let’s say we are interested in finding all the romantic restaurants in New York City right.

David Kruse: Yeah.

Jeff Palmucci: So you know we have many hundreds of – many millions of users that you know fill out these – fill out their review forms, they give us reviews on all the restaurants and attractions or whatever in New York City. Well at the end we’ll ask a couple of yes/no questions. You know is this a romantic restaurant? Is this a – if this is a family friendly hotel? You know whatever we are interested in surfacing that we think people are interesting in looking for. And then we can use that data as a label set in order to predict for a new locations whether those relationships don’t apply. So that’s a very powerful way to get some very generic groupings of places on site, right. So if I’m going out for a data I can find some romantic event. Like I have a bunch of kids they can find some kids friendly, but there is – you know that’s pretty much straightforward natural language direction that we use for that. Some of the interesting stuff that we are working on now is some of the keyword conversation stuff. So if you go to a restaurant on the site and scroll down just to go to the views, you get a bunch of key words that we identify. And those key words are meaningful, you know so there is some learning work that goes up to deciding where Central Station really isn’t a thing in New York City, the word that you are looking for is Grand Central Station, right. So trying to identify phrase that as meaning or conceptual meaning. And then once you identify those you can do all sorts of statistics on them, right. So what is this particular restaurant know for. Well let’s find the concepts, let surface more options on this restaurant than the streaming ones. What is New York know for, right. Well Broadway pops up right, you know where there is Broadway shows. So you know just simple – once you identify these concepts, just simple counting you know we used to Spark to count over our millions of reviews, so it goes pretty quick. It can get you a lot of the interesting things to surface. So yes, check out of the sign, check out the view, well it’s a pretty interesting way to explore. Another thing we are working on now is grouping places together, right. So you can find, if there is work direct, it’s a way to translate – there is way to translate words into a vector which has meaning in it and we are using a derivative of that to try and group places together. So we can look at common themes in the various different cities and try to surface those. It’s really interesting stuff and you know…

David Kruse: That’s good yeah, that sounds tough, but it sounds – so are you…

Jeff Palmucci: Actually there is a lot of the stuff is written about it in our blog enduring at TripAdvisor.com. In fact the last two projects that I’ve talked about are written about there, people are interested in checking out.

David Kruse: Great, that’s good. We should put a link to that on our site and so what are these models, because you have – people are continually rating and adding comments, so they are adaptive models you have underneath or…

Jeff Palmucci: Oh! Yeah.

David Kruse: They are. Okay.

Jeff Palmucci: So like when we will first train a model you know we’ll typically have very few labeled points and we’ll use the model – we won’t use the model to actually label stuff on the site, but we’ll use the model to guide where to ask the questions next. Let’s take a quick example, I hope I’m not going over too much. But let’s take a quick example. Let say I want to find all the leading restaurants in the world, okay. How many leading restaurants do you think there are? There is not that many, so you can’t just randomly start asking hey, is this a big restaurant, you know to every user you know? Hey, is this a modern stake house, is this is a leading restaurant. You know will get a bunch of no’s, bad data to train on right, and you will look like an idiot to your users. And so what you want to do is guide the questions, right. So we’ll search though all the reviews for the word weekend and we’ll start asking the leading questions on those places and then we will build a model using that data and then using that model roll out where we ask questions down the road. So the model may notice a correlation between Hannis and the place being a leading restaurant and then we’ll start asking on places that have the world Hannis in their review text a lot. And in that way you could have the machine learning model kind of creep outwards to find all the leading restaurants in the world. So it’s kind of an interesting process. And you know its wholly crowd driven, right, so you know we don’t have to put together any material trackers or anything like that. Our users not only there are the much more than and they are very, very helpful, but they also know what they are talking about, right. So if you are filling out a review for a particular restaurant and you ask that person hey, is this is a leading restaurant, well they have just been there so they know, right. Whereas if you actually try to get – collect this data online, you are going to be doing your research on web and in fact that’s kind of where probably you haven’t actually been there. So kind of an interesting way that you can use crowd source data to input all sorts of interesting stuff about location centers.

David Kruse: I think a lot of machine learning researchers who were probably a little bit jealous on the amount of the data you have that’s…

Jeff Palmucci: Well, they should give me a call, because we are hiring.

David Kruse: That’s right, there it is, that’s good. And yeah so, how many – I imagine you have lots of models. I think you mentioned you know hundreds. So how do you know – what algorithms do you use and which ones, and how do you know which ones to use based on what problem?

Jeff Palmucci: Okay, so one of the things about the group here is that we try to be very pragmatic and the models that we use are going to be the models that work right. So we are not really tied to any particular product or anything like that. So the classification there is talking about just, that’s already in a spark and I tried to use Amoled for a while, but it just didn’t quite get the problems that I want to put in my own kind of creating into the sent optimizer, but you know that was a particular solution that we needed to write because we are dealing with millions and millions of reviews right. But we’ll also use R and so I could learn to do models that can fit in memory. We will also use elastic search for kind of a network analysis. You know we’ll basically use what works right. We are never going to merge any particular technology.

David Kruse: Got you, within our another – I don’t know if you use Python or another language, you know how do you know which algorithms to even start with based on. I guess just classification algorithms, I suppose you can start there, yeah.

Jeff Palmucci: Well yeah, you know you got to feel for these things as you know from experience right. So if you are trying to build a classification model and there is lots of non-linearities in there, maybe use a sport victor machine with an RBF carnal or something like that. If you think it’s a pretty straight forward model and if you want to really understand what’s going on maybe use a logistic aggression. You know there is some models that – sometimes you will need a deeper model. Like there is one example of this person here that built a model to try and identify circumvention. So if you are, if you are on a vacation rental and you are listing, you are listing somebody’s property, you want to make sure that they are not trying to get in the system and you know get out of paying the fees if we actually get them the sale. So we actually built a deep learning character model, which is kind of an interesting approach. In that you know the deep, the features of the deep learning network learnt were words, right, but there were all sorts of different way that people would try to gain the words so that it would get past the regular expression filters that we used and we’ve got a lot of good success on that. And of course you know the deep learning is very good for the vision systems that we are playing around with now too.

David Kruse: What vision system? You said vision systems?

Jeff Palmucci: Yeah, yeah so I mean we got…

David Kruse: Yeah, what are you working on in that space?

Jeff Palmucci: So, we don’t really have anything in production yet, but we are playing with very interesting stuff like you know – we have all these restaurants online, find all the menus so that we can make sure that you can get to them before going to the place, you can check out what they have. If you are doing a filtered search, so let’s say I’m searching for sushi in New York, right, what kind of pictures do I want to show to the users? Do I want to show just the random pictures of the different restaurant or should I want to show the sushi; that’s something that we are going to try out.

David Kruse: Interesting.

Jeff Palmucci: And then there is just quality. You know there is a lot of millions and millions of places that are listed and sometimes we don’t pick the best photos. We do a lot by hand, but the volume is just too great sometimes to get good listing photos for everything, so we are working on a model to try that too, yeah.

David Kruse: Cool and I think we are coming to the – near the end of the interview here unfortunately, but I do, I think I read that you are photographer, is that right?

Jeff Palmucci: Yeah, yeah I’m a concept photographer. I have been doing it for several years now, yeah. So I shoot for New England Music News and Theywillrockyou.com.

David Kruse: Are you serious, really.

Jeff Palmucci: Yeah, yeah so mostly harder rock, although tomorrow I’m shooting Jack Burge at the House of Blues in Boston. So yeah, so it’s a lot of fun. I got into photography many years ago and a friend of mine who was a concept photographer invited me up along a couple of days and it just kind of went from there, and [inaudible] and it’s kind of fun. You know photography is an excellent hobby for nerds because of the electronics. You can really – if you give up the electronics and all the image processing stuff and still satisfy an artistic scratch, you know.

David Kruse: Yeah, definitely. Have you photographed any super memorable concerts that you come up that you remember?

Jeff Palmucci: Let’s see, I have done a lot. So I have done – so I do a lot of metal bands right. So we are allowed to – so does this – there is this festival that goes on in part of Oklahoma, it’s usually called Rockoklahoma, and we have done that a several times, we just road tripped out there and it’s just a lot of fun as a concert because everybody there is really freaking dedicated to having a good time. So you know, I shot Green Day and had them like lean over the stage with this guitar. I mean it’s you know it’s – yeah it. I shot some of my idols from when I was kid. I shot Judas Priest right when they came around yeah. So its fun, I think I’ve got a pretty fun hobby and I’m kind of looking to fill into it.

David Kruse: Do you – and this is probably a scratch, but do you think that hobby helps at all with machine learning?

Jeff Palmucci: You know I would like to say that all the stuff that I know about image processing helps me with the machine vision. The truth is I don’t know what the hell is going on inside these networks you know. You just string them together and wow! It kind of works, and that’s kind of cool, so yeah.

David Kruse: And that’s the beauty and kind of the issue of keep learning, but okay, interesting. Well, let’s see. I think that kind of much probably does it. I guess well, I am curious to – sometimes I ask the interviewees, do you have – you know when you look at in your career, any major mistakes. Any life lessons that you can think of?

Jeff Palmucci: Yeah, I could think of one. Sometimes you have people that are working out for the team, and I think the instinct for anybody is try and make it work out as hard as you can and you know sometimes that just isn’t the best thing for the team. So you there were a couple of mistakes that I made where I was too slow to take action on it, so I thought I wasn’t doing well and you know I think I kind of hurt the team as a whole. Fortunately nothing here, so.

David Kruse: That’s good, live and learn. So hopefully we can learn, now we can learn from you.

Jeff Palmucci: Yeah, yeah. But you know sometimes you just got to do it the painful way, so .

David Kruse: Nice, all right well, this has been great. I definitely appreciate your time Jeff and to hear about what you have done and what you are doing. So it’s pretty fascinating.

Jeff Palmucci: Great, well thanks for having that sent back to me.

David Kruse: Definitely and thanks everyone for listening to another episode of Flyover Labs. As always, I greatly appreciate it and we’ll see you next time. Bye everyone. Thanks Jeff.

Jeff Palmucci: Okay, thank you.