#14 Christoph Bodenstein – Product Manager at Adverity on building AI products for marketing technologists

Impassionate product manager in an agile, B2B SaaS MarTech & data company. Profound background in Data-Driven Advertising, Marketing Analytics and Business Intelligence. Curiously following web-, tech- and new media-related trends. Creative thinker. Digital native. An admirer of beautiful designs. Technology lover. Early adopter. Challenger of status quo’s.

#14 Christoph Bodenstein – Product Manager at Adverity on building AI products for marketing technologists Get Your AI On!

Ciprian Borodescu: I’m here with Christoph Bodenstein, Product Manager at Adverity – a marketing technology platform that enables brands and agencies to make better marketing decisions by integrating all data into a single source of truth and leveraging artificial intelligence to uncover opportunities. I’m super excited to have you on this podcast. Thank you so much for being here.

Christoph Bodenstein: Ciprian, honestly, I’m more than delighted you’re having me today. Thanks for the invitation.

Ciprian Borodescu: Awesome. Tell us a bit about yourself, your background, and why not describe a day in your life as a Product Manager?

Christoph Bodenstein: Alright, sure. So I’m based in Vienna. I’ve studied economics here in Vienna and after that, I started working for a digital performance marketing agency. And from this point on, digital marketing has kind of become my big passion since then, I would say. I had the opportunity to manage and consult very large accounts and had the opportunity to spend large budgets on campaigns. Everything we did was very tech-driven, meaning that every euro we spent was exactly tracked, and its influence on the business success has been evaluated. It was super exciting. I learned a lot. And, as time has passed, I moved more and more into the technological side of data-driven advertising. I started writing scripts, to automate tasks, and work on simple algorithms that optimized campaigns automatically. And after a year, the agency has started using a tool to automate the marketing reporting. So, you kind of have to imagine that at this time, every person was spending like 30 minutes a day to manually download data from advertising systems, such as Google Ads, or Facebook Ads, and paste them into huge, clunky Excel sheets. It took them, like, five minutes, just to open up the sheet because they were so comprehensive, and after everything was pasted, they typed it manually into an email and sent it to 10 people that never really opened it up. And this tool, we started to automate this entire process, it was called Adverity. And, yeah, I generally love to try out every sort of tool and Adverity immediately grabbed my attention. I really nerded into it and was very excited to automate every manual process possible. And yes, as fate would have it, I ultimately ended up working for Adverity. They’re also based in Vienna, so it was a perfect fit. At that time, I also knew the user perspective very well. I started in product management, where I still am, and currently, I’m responsible for regularly talking to users, find out what capabilities they’re missing in a platform, and discuss with our data science team and engineering team how to best solve these problems and ship new features.

Ciprian Borodescu: And recently, you launched a new AI product part of the Adverity platform. Can you please share more about that?

Christoph Bodenstein: Sure. So yeah, to fully understand the value of the feature that we are building, I have to go back a little further to explain the evolution in marketing analytics in the last few decades in general, if that’s okay.

Ciprian Borodescu: Okay.

Christoph Bodenstein: So some years ago, marketing analytics was a very specialist and manual coding exercise or as I described before from the agency use case, a manual copy-pasting Excel game. And to obtain intelligence from data, very time-consuming and detailed tasks were needed. So you know that large advertising sources are tracking hundreds of KPIs, metrics, and segmentation possibilities, and also providing them to the campaign managers. And also, therefore, every campaign manager that starts a campaign inevitably starts all the process of enormous data generation. And this data also needs to be analyzed constantly. However, the problem was that those insights that campaign managers had from their manual evaluation were so late in emerging, they had become irrelevant already again. So in more recent times, we have now seen the arrival of many visualization and business intelligence platforms that are also very specialized into analyzing marketing data. They’re supposed to harmonize data from the resources and create graphical charts and dashboards to track KPIs in an automated manner. And while this was a major step forward already, it still requires a reactive approach of manually interrogating those dashboards to search for the hidden nuggets of intelligence. And this can take many days; plus, there’s no guarantee all of the valuable insights will be found at all.

Christoph Bodenstein: So what we have now built is a so-called augmented analytics platform. The idea is that we use some machine learning techniques to analyze huge amounts of marketing data constantly, use self-organizing decision trees to classify datasets, and search for insights automatically. These insights can be detected patterns in your data, anomalies, or optimization recommendations based on self-learning regression models. Ultimately, each insight we detect is ranked in terms of criticality, and the user gets proactively informed about relevant developments in their marketing data. So, it uncovers deep insights, which would otherwise never be found amongst a huge volume of information available, it has a high level of automation to the analysis process. It’s completely out of the box. Plus, it makes recommendations proactively on where to best focus resources.

Ciprian Borodescu: That’s super interesting. And I am wondering, from a product point of view, what were the main challenges in building this product? And how did you overcome them? And I’m referring here both to the technical and business challenges.

Christoph Bodenstein: Yeah, so there’s some of them that immediately pop into my mind. But I would say the most worth mentioning is I think the intersection of three very complex knowledge areas that had to be combined here – and still have to be combined. So, on the one hand, we have performance-driven advertising that requires domain knowledge of the advertising tool itself, that requires knowledge on how to reach specific goals on certain platforms and knowledge of how to optimize some campaigns. Then we have data science. So normally, marketeers are not that number-driven, I would say, and what we’re trying to do is to give marketeers a framework to answer questions more rational, more data-science backed, more statistically backed. And so, the second part is data science, where data scientists first always have to understand the business value of everything we do. Then, they come up with researching different packages, different libraries, they’re coming up with algorithms. And then, I have to understand the algorithms that they build so I can communicate them. And of course, the third section is clashing together with engineering. I have no development background. And so, it’s still sometimes hard to catch up with everything, but it’s getting better.

Christoph Bodenstein: And yeah, how did we overcome them? I think the key is always some communication. So, 80% of the team is working remotely; during the current time, it’s almost 100%, I would say. We’re updating ourselves within the team daily about business, commercial, and technological-related developments about achievements from data scientists, we are presenting regularly libraries and packages and algorithms researched, evaluated, and tested for specific use cases. And I always found it very important to involve as many people in as many decisions as possible. So, kind of everyone knows what is going on for other people in the team, at least at a high level.

Ciprian Borodescu: I think that’s a good mindset. I’ve had other guests on the show, and everybody seems to be on the same page when building AI teams. Everybody was mentioning the same thing you mentioned, the fact that you need a cross-functional team. They need to have domain expertise, they also have to have engineering, and somebody to tie it all together. And I guess you were that person in this case.

Christoph Bodenstein: Yeah. You could see it like that. So, what I’m describing here is the team that’s set up for especially this feature that we’ve now recently launched. The entire team is, of course, way bigger. But yeah, we can talk about how they’re structured in detail, a bit later.

Ciprian Borodescu: Where did you have the biggest challenge? Or was it challenging to talk to engineers and to describe the business problems to engineers? Or was it marketers trying to understand the kind of product you’re trying to build for them.

Christoph Bodenstein: Yeah, actually both parts. So, we’re at the intersection of tech and marketing and we’re having a very education-heavy product, I think. So, we have to constantly invest hours into explaining what we’re actually doing, not only to customers but also internally, so everyone is constantly on the same page. Also, something that might have fallen short in the beginning, but after we invested more time into education sessions for everyone, everything has become better. So, engineers were able to understand business values by themselves and started to think about features by themselves, but also sales and the commercial team had a far better framework to pitch our solution to the right audience, and I had more time to focus on other things rather than explaining to individual stakeholders.

Ciprian Borodescu: On actually doing the things, right? Building the product.

Christoph Bodenstein: Yeah, exactly. There was also, actually, another challenge that we had, which is more tech-driven, I would say. Especially in the beginning, we underestimated the importance of doing investment into building a black box. So we’re building a practical analytics solution that only notifies users on development, if there are any, but even more important than analyzing the insights that we detect is analyzing the insights we dismissed – so the ones we did not show and we’re building in a very feature-driven and customer-focused way. And these investments in tracking mechanisms sometimes fall by the wayside as they do not create an immediate value for the customers. And there was a point where we couldn’t tell why the computation has decided in that way, why it has shown this specific insight, and why not. And so, this was also a major step forward, that we invested some months into technical investments to make this more transparent, even though this had no direct impact on customer benefits, but it benefited us greatly to evaluate our algorithms better.

Ciprian Borodescu: Yeah. It makes a lot of sense. Yeah. Okay. Let’s talk about some of the mistakes you’ve made along the way and perhaps what were the key learnings there? And if you’d start all over again, what would you change?

Christoph Bodenstein: I think that’s to invest in a better educational framework for everyone, not only for engineers, not only for customers but for sales, dedicatedly targeted to different target audiences, what our algorithms are doing, what are they solving? And how can users benefit from it?

Ciprian Borodescu: What are some of the most important roles you believe a product team should consist of, especially if it’s an AI company at the core? And maybe talk a bit about these teams as relative to the stage of the company from startup to a scale-up company the size of Adverity.

Christoph Bodenstein: Okay, so when I started at Adverity, we had like 60 employees, 40 of them were engineers. Today we’re 181 with 85 engineers. So, I’ve gone through some organizational approaches by now. Currently, we’re structured, or at least the development team is structured in so-called pots. These are small, agile, cross-functional, and autonomous teams responsible for certain areas of the product. Each pot consists of core members, which are engineers led by a pot owner, who is also an engineer, and the product manager. And furthermore, it has some assigned specialists such as data scientists and designers, or tech advisors for certain domain knowledge areas. In the product team itself, we are a very diverse set of people with different backgrounds. So, I have more of a marketing background and I have been a power user myself, which makes it easy to kind of see the product through the eyes of the user. But, of course, I have other amazing colleagues with other knowledge areas; some of them have a more technical background and know the marketing APIs inside out. Some of them have a B2C PM background and some of them were transferring from other departments such as account management. And therefore, it was always very beneficial for everyone to exchange as much as possible with such a diverse team. So basically, everyone in the product team is assigned to different pots that I just mentioned before, depending on their expertise, their experience, and of course, where they want to be. But of course, it is also very dynamic and changes quite often. So, all in all, I would say the key is that it’s a well-diversified team with a well-balanced amount of backgrounds as well, to have a good product team.

Ciprian Borodescu: Okay. And talking about diversity, where do you see things heading? Because, personally, I feel like there are more and more women involved in STEM – which is amazing, of course; however, it still feels that we need to be intentional about it and be on the lookout for women that are interested in a technical career. And I’m wondering, based on your experience, how the ecosystem feels in Central Europe versus Eastern Europe, in terms of diversity in the AI talent pool?

Christoph Bodenstein: So, what’s interesting is that the majority of our tech force at Adverity is actually either working from Eastern Europe remotely or moved to Vienna from there, so we are kind of very Eastern Europe dominated. So we might have the same perception of the ecosystem. Our engineering team is currently clearly dominated by men. In July, we hired our first female data scientist who is from Russia. Apart from that, I think we have three other female engineers. My personal perception is that the numbers of women in tech is rising, also, at Adverity. And you mentioned it already, I truly believe and I have also experienced it myself, and I think everyone agrees that balanced teams are more creative and have better problem-solving capabilities. I already mentioned it with the ideal product team. So, it would be beneficial for everyone to have more women in data science and engineering. To be honest, I always find it a bit presumptuous to give a woman advice as a man, as I maybe do not know what they’re going through. But what I do think definitely helps is simply encouraging them. I have the impression there’s a lot of competition in the tech world and if you just support them, they’re going to be way more likely to succeed and stick with a career in the tech space. We have some female engineers at Adverity who are also actively supporting female developer initiatives, and I think that’s great. I think that really helps in establishing female coding communities, and we need more of that.

Ciprian Borodescu: Yeah, yeah, absolutely. And now, clearly, AI is hyped these days, and I’d like to invite you to give our listeners a framework or a set of questions that can help them identify those companies or startups that are indeed doing AI versus those that just play around with some keywords on a landing page.

Christoph Bodenstein: Yeah, that’s a great question. What I always do in order to find out if a company is actually using AI, is to understand what they’re actually doing. Surprise. It seems we’re not only in the age of companies advertising they’re doing AI but also in the age of companies not advertising at all what they actually do. I’m frequently visiting tech vendor websites and after reading through their websites, I still have no clue what they do. The websites are often full of passwords and with no real content, and I think this is the first step of evaluating a company’s AI capabilities. So, maybe after receiving a demo, or reading some customer reviews, or analyzing UI screenshots, you can actually also better understand if this is something AI is actually needed for. What kind of data do they have to process in the background in order to make the product work? How much data do they have to process? Can it be solved with simple arithmetic functions or database queries? And after you can answer these questions, I think you can also better understand what the company would need AI for in order to solve specific problems. Or – this is also something I’m asked frequently as a PM that is in contact with users – just go ahead and ask them what they actually use AI for. Of course, what you need to notice here is that this only works if you have a certain domain expertise in a specific field. So I think I could tell which marketing tech company might be really using AI after analyzing it, but I definitely couldn’t tell for a biotech company, for example.

Ciprian Borodescu: Absolutely. Yeah. And I think one other thing that can be used in analyzing if that company is actually producing AI models or using AI capabilities to see if they actually have data scientists or machine learning engineers part of their team or if they publish research papers and stuff because that’s something that a lot of machine learning engineers are doing.

Christoph Bodenstein: Yeah, I just think that the position of a data scientist also has become very widely spent already, so it can mean a lot, so you also have to be careful with that.

Ciprian Borodescu: Absolutely. Okay. So what’s your plan for the rest of the year? And what are the growth opportunities for your company as we move out of this pandemic, hopefully, next year?

Christoph Bodenstein: Yeah, so it’s hard to plan in these times. I’m turning 30 this year. For Adverity, my clear focus is to make our newly launched AI module kind of the industry standard for augmented analytics, make it also the new normal to analyze marketing data. I find it super amazing to work on such innovative technology. Personally, I really hope that we can return to the office in bigger numbers soon. I’m maybe a bit old-fashioned there, but I really love to brainstorm and exchange with other people physically.

Ciprian Borodescu: I totally get that. Yeah, I’m also like that.

Christoph Bodenstein: Okay. And yeah. Furthermore, I’m looking forward to many new colleagues that might join us throughout the year. And personally, of course, I’m on the hunt of learning something new every day as it is the moment and what I really enjoy.

Ciprian Borodescu: Okay. Alright. Awesome. So, for the final special section on the podcast, lightning questions and answers, a series of fun short questions that you have to answer really, really fast. Are you ready?

Christoph Bodenstein: Okay.

Ciprian Borodescu: Game of Thrones or Lord of the Rings?

Christoph Bodenstein: Game of Thrones.

Ciprian Borodescu: Okay. Star Wars or Star Trek?

Christoph Bodenstein: Star Wars.

Ciprian Borodescu: Star Wars. Okay. I’m not gonna comment on that. Favorite movie?

Christoph Bodenstein: That’s a tough one. There are so many.

Ciprian Borodescu: Let’s go maybe with the last one you watched on Netflix because everybody’s doing that nowadays. Right?

Christoph Bodenstein: The last one on Netflix. So yesterday, I watched Tenet by Christopher Nolan in the cinema, and that was pretty mind-bending. Have you heard of it?

Ciprian Borodescu: No. No, but I got the recommendation. Yeah. I’m surprised that you said on cinema. Do you guys actually have cinemas open over there?

Christoph Bodenstein: Yes, we do now. So, they opened up this week again.

Ciprian Borodescu: That’s why. There we go.

Christoph Bodenstein: And it’s from Christopher Nolan who also did Inception, and he had, like, a crazy new movie that’s very hard to follow. That is awesome.

Ciprian Borodescu: Awesome. Thank you for the recommendation, I’m gonna check it out. Awesome. I loved Inception. I really did. Cats or dogs? Come on. This is easy, right?

Christoph Bodenstein: For me, it’s cats.

Ciprian Borodescu: Cats. Perfect. For me too. The last book you read.

Christoph Bodenstein: To be honest, in today’s world, there’s so much media to consume, so I have to admit that I only find time to read books in vacations and my last vacation was just before the lockdown in Kenya, and there I’ve read Yuval Harari’s 21 Lessons for the 21st Century. It’s a great book.

Ciprian Borodescu: Oh, nice. Nice. Thank you for that. Okay, so, the bonus question: marketers or engineers?

Christoph Bodenstein: A diplomatic answer: the intersection of both and that might be the most interesting of all areas.

Ciprian Borodescu: Okay, do you have a name, by the way, for the intersection of marketers and engineers? Is there a name, a role?

Christoph Bodenstein: A role not really but in the industry is called edtech or martech. But to have one specific role that is, I don’t know, whatever doing in this industry, I think it doesn’t really exist.

Ciprian Borodescu: Yeah. Maybe marketing technologist or something like that. I don’t know. Awesome. Well, Chris, thank you so much. It was a pleasure to have you, and thank you so much for sharing your wisdom with me and with us. How can people reach out to you for ideas or comments?

Christoph Bodenstein: Well, the best would be via LinkedIn. Simply search for my name, connect with me or shoot me a message. And thanks for having me, Ciprian.