AI Risk, Regulated Industries & Leading Without Answers

Victoria Mensch: Hello and welcome to Decisions at the Frontier, brought to you by Silicon Valley Executive Academy, where we help leaders tap into the Silicon Valley Innovation Playbook. My guest today has lived through more than one big technology wave — the move to cloud, to IoT, to enterprise data platforms. Now he's building AI systems inside one of the most consequential environments there is for getting it right. And he's going to tell us why he thinks this wave is different from all the ones that came before it. Welcome, Ahmad Fattahi, the executive director of generative AI at Genentech. Welcome to Decisions at the Frontier, Ahmad.

Victoria Mensch: As I mentioned, you've lived through more than one of the big technology waves, and when we chatted before this conversation you mentioned that AI is different. People call those waves transformative as well. Why do you think this one is actually different, and not just louder?

Ahmad Fattahi: It's a great question. There is no certainty yet, but there's enough evidence to be confident that this wave is different. Why? A couple of reasons.

Ahmad Fattahi: One of them is the already evident impacts on both efficiency and effectiveness. Recently a Nobel Prize in Chemistry was won by very capable humans who in large part leveraged AI — arguably you can assume the Nobel Prize was won by a combination of very smart humans and AI. So at the highest levels of scrutiny, scientific scrutiny, there has been evidence of this AI wave being different, solving very hard problems. Every day you hear from the world of math and algorithmic design that new solutions are being offered by AI — OpenAI recently announced that ten problems were solved by AI. Just this morning, on the news, researchers from Stanford announced that some viruses were designed with the help of generative AI, in that case to help fight other harmful living entities.

Ahmad Fattahi: That's one significant difference from the former waves of AI. If you go back a few decades there have been a few waves where people thought, 'this is the time.' If you go back to the expert systems of the 1980s, for example, there was a short-lived wave that went away. That's one big reason. From the economic angle it's also significantly different: the amount of both investment and already-harvested benefits is not parallel to anything we have seen before, and most of this information is public. You can look at, for example, how much Google Cloud is relying on generative AI services. The economic impact is real and is happening. There are still remaining questions about whether the benefits are on par with the massive amount of investment, and the jury is still out. But given the nature of the problems we are already seeing resolved by generative AI, as well as the economic facts on the ground, those two alone are enough for me to say that this wave is different.

Victoria Mensch: If you look at the previous waves of technology — and we kind of operate in this wave here in Silicon Valley — there was cloud, there was mobility, there was big data and digital transformation migrations. I'm sure they also supported the development of new solutions. What is it, specifically about AI, that changes the risk calculation for leaders? What do we need to pay much more attention to than with those previous waves?

Ahmad Fattahi: That's a great question. Many of the symptoms or topics you start from under this umbrella of risk — sometimes people ask, 'what keeps you awake at night?' — many of them eventually lead to autonomy, and to the amount of cognitive and decision authority that this time humans are giving up, sometimes knowingly, sometimes unknowingly. If you sit back for a second, we're actually giving up more and more to AI. The analogy I often use, and it resonates with a lot of people, is to look at the business of going from point A to point B. Forty or fifty years ago we humans used to look at our destination, open a physical map, chart our route, and either memorize it or have somebody sit next to us and tell us what to do. So humans were basically in charge of both finding the route and executing while we drove. Then maybe around the year 2000 — some of us will remember — Yahoo Maps or MapQuest.

Ahmad Fattahi: So we started to put the destination into those pieces of software. It gave us the route. Sometimes we did a sanity check before starting, but we gave up that part, the routing, and we just looked at the map and started to execute. Around 2010 most people started to give up a little bit more and say, 'hey, TomTom or Google Maps on my phone, this is my destination, go,' and we didn't check anything. We just took the execution wheel: if it said turn right, we turned right; if it said stop, we stopped. But we still were the executioner. Now with the self-driving car we are giving up the last remaining piece to the machine, and we're literally sitting in the back seat.

Ahmad Fattahi: So over the course of a few decades we started from doing everything, and bit by bit we gave up both the decision power — the autonomy — and the execution of this whole business. In the near future everybody expects it will take over more and more of even our individual consumer commute as well. If you expand that example to how we write email and how we consume news — sometimes it's invisible, we don't think about it — the very important aspect of receiving information every day in the news is absolutely automated. It's decided by AI, which is a very important decision for anybody's day-to-day or even long-term life. It's essentially all decided by AI, 95% at least. We can choose from time to time to not use this app but that app, but how often does that happen? I have my own routine every day and it's fully decided by AI what I'm listening to and not. You can expand that to what we eat, what we consume, what we buy.

Ahmad Fattahi: So what's different — to go back to your question, full circle — is that the cloud revolution of the 2000s, around 2010 to 2012, was mostly a technological advancement. It was this massive thing: 'hey, you don't need a data center here or a bunch of computers right there; there's this invisible thing, a bunch of computers sitting somewhere, doing all of your calculations or storage for you.' But eventually it was similar in nature. You were still in control; it was just a different technological setup. What's fundamentally different this time is that they're being autonomous. These agents in AI are taking over cognition and decision-making, which is fundamentally different from the previous waves. And I can't wait to see what happens when the same level of revolution happens to physical AI and the robots — that will be, in my opinion, the next wave of significant leap.

Victoria Mensch: Yes — I think what you're talking about is really that giving up of control over judgment, right? In decision-making. I absolutely agree with you that once AI starts coming from a digital world into the physical world, that question of trust and the judgment call becomes even more critical and has much wider implications, especially from a business point of view, because the impact of making the wrong judgment and the wrong decision is much larger. So given all that, when you are designing these systems for organizations and businesses, how do you account for this? What should business professionals and executives look out for?

Ahmad Fattahi: Very good question, and a hard one actually. There are multiple layers to consider. The general mindset should be that there is no absolute immunity. I'll offer a few layers that in my experience are very helpful and effective, but nothing is going to be 100% bulletproof. After saying and doing all of these steps, there has to be a layer of: okay, what should we prepare for in case every layer fails? That's a given — an age-old principle from cybersecurity, and the same mentality applies here: things like kill switches and contingency plans. Nothing that I'm going to offer is a replacement for those. Now, to the layers. It starts from data.

Ahmad Fattahi: Your data has to be very intentionally opened up — privileged access. The same principle that applies to humans applies to agents: the best practice of giving access to data to people based on actual need, as opposed to opening it up to everybody with full access. Another layer is when you build the application with AI. There are things you can instruct it to do both in a hard way in the software — certain operations are programmatically blocked — and in a softer way, when you instruct generative AI models to behave in a certain way, or not to do certain things, or to avoid, for example, engaging in politics if that's applicable to your application. There are softer ways to instruct your model, and it has a big impact. You can also add another layer: a different model, agent or tool can come after the creator — the generative model — and judge it, to make sure independently that it actually meets the criteria.

Ahmad Fattahi: That's not a bad idea, and usually people want to use different models between the two, so it's not the same model creating and judging itself. There are techniques like this that can be very helpful. Another important factor is that in highly sensitive use cases you either want an expert human to always be in the loop, or at least be ready for escalated cases. It obviously varies: if it's a low-sensitivity application — picking from a number of news sources, to go back to my previous example — it may not be that sensitive, so it's okay to not have a human. You also need to consider cost and scalability. But at the other end of the spectrum, if it's a model that's supposed to prescribe medication to patients or make diagnostics, those are highly sensitive and regulated for good reasons — applications where you need to dial up the need for a human to be in the loop, and not any human, an expert human.

Ahmad Fattahi: There are applications in the middle where sometimes it's not justified to have a human always in the loop, but you can design and build layers that on the fly make the judgment that this case is good, but the next case needs to be escalated to a human to review. Some insurance cases are like that: if the picture and the details of the case are pretty obvious, and the AI and the follow-on layers are pretty confident that everything aligns, they can proceed on their own. If there's a discrepancy — if the photo of the damage and the descriptions don't align exactly — by AI judgment they can escalate that to the human. So these are a few techniques. There's a lot more, but these are a few high-level techniques that are quite effective.

Victoria Mensch: That's very useful — basically layers to mitigate risk, if you think of it. Speaking of risk: a lot of AI promises 'move fast, iterate, ship,' and I think overall that is the Silicon Valley culture, right? Move fast, fail fast. In some environments it works, but in other environments, as you mentioned, that are more sensitive, you probably can't just ship and see what happens. There's a validation burden, there's a documentation burden. What does that actually do to how you think about AI development compared to environments that don't really have this constraint — where risk tolerance is much higher in some environments and much lower in others?

Victoria Mensch: And when you're dealing with a different risk profile — or trust profile, in a sense — the consequences are not the same in those more regulated, less risk-tolerant environments.

Ahmad Fattahi: Sure. If you do it right, even in those less risky environments — the way I generically think about this in my head, at what speed and at what level of risk you should move, even in the less risky environments — it boils down to the reward versus the risk. Even in those environments where you say fail fast or move fast, it's not black or white. There's a cost to everything, every failure. As long as, beforehand, before taking the risk, it's clear why we are taking this risk, and you, to the best of your capacity as a decision maker and as a team, imagine: if we succeed, X happens — we have won this knowledge or market or whatever it may be. And if we lose, what's the worst that can happen? And arguably more importantly, what would we have learned out of the failure, and what would be our next step? As opposed to blindly saying, 'oh, we need to move fast, we don't care, it's bad to not do anything, let's just move.'

Ahmad Fattahi: That latter scenario can be pretty hurtful economically if you're in a less regulated and less risky environment. It can be catastrophic if you're in a highly regulated environment. So with that 'not black or white, but what shade of gray' mentality: if you're in a more regulated environment, your calculation changes toward 'okay, a failure can exert a much higher cost, therefore I need to measure three times before I cut; I need to have five more conversations with human experts and understand even more deeply what happens if this piece of software, this AI, makes a wrong decision; when it would be good to escalate to humans.' Whereas if it's a less regulated, less risky environment, then maybe much earlier in the process you say, 'okay, it's good enough,' because the cost of failure is much lower. It changes your calculation that way. I pretty strongly believe it's not black or white. It's what shade of gray.

Ahmad Fattahi: And there are many different industries — even within an industry. I come from the pharmaceutical industry and I've been exposed to several other industries in my previous jobs. Even within pharmaceuticals, it's one thing to say, 'hey, human being, this is your prescription, your drug, or your diagnosis' — that's obviously very sensitive, highly regulated — as opposed to working on an internal operational application that helps a small group of people do their job 30% faster without external exposure. I'm not saying you don't have to be careful, but the level of risk is very different.

Victoria Mensch: Speaking of different industries, Ahmad — you mentioned that you have experience across enterprise and networking in telecommunications, and pharmaceuticals. What worked in your experience? Maybe there's something that worked fantastically in those other industries and doesn't work for the pharmaceutical industry, and vice versa.

Ahmad Fattahi: Some of it goes back to your point about the level of risk and regulation. Certain things — for example, in manufacturing, automating a process, having an AI at a certain part of your manufacturing process — can deliver, if it's done right, a very significant amount of efficiency gain to your production. If you are in the business of high-tech manufacturing — chips, especially given the size of this market now with the need for memory and GPUs and all that because of AI — a mere 1% improvement in the yield by the end of your manufacturing process can be really consequential for your bottom line. And the risk, on the other hand, assuming that you have testing and verification afterwards anyway, may not be as high to try something earlier; the bar to clear is lower.

Ahmad Fattahi: Compare that with an externally facing, patient-facing AI that is put out to have a conversation with patients, either in chat or through voice. Making one false statement, or even one incomplete statement, in the US per FDA regulations can have big consequences, putting both lives and the company's reputation and bottom line at risk. There are absolute penalties for not making the declarations and statements that need to be made in certain cases; there are penalties for not reporting back if a patient reports an adverse event, for example. None of these sensitive aspects are applicable to a manufacturing environment in my example.

Ahmad Fattahi: So bottom line, you can't just say 'oh, AI, smart, I'm just throwing it in.' There's still a big piece of responsibility on the human decision maker to have the expertise of the industry and the use case before they decide that this AI would fit with these kinds of guardrails, and that otherwise other cases wouldn't make sense. Which brings us to a bigger point. People talk about what AI is going to do to the job market and skills. One of the scenarios that I happen to subscribe to is that, at least for the next few years, a big part of what remains highly sought after and required — arguably even more so — is the expertise that humans do have: this broader knowledge of the world, and much more deeply in that specific industry. Imagine the calculation that goes through an executive's head before making a decision that encompasses the finances, the process, the workforce morale, the reputation of the company, the regulatory environment — all of that, the result of 20 years of experience living through this life. AI is not there yet. It's very good at relatively point solutions.

Ahmad Fattahi: So injecting the expertise is still very heavily on the humans. In three years, five years, whether that's going to change — we don't know — but that's where we are.

Victoria Mensch: That's an interesting point, Ahmad, and it's something quite frankly we're discussing a lot in our programs and with executives from other companies as well. There is an understanding of the value of the expertise as it exists right now, the way humans accumulated it. Now the question is how it can be captured — first of all, can it be captured? — and then what about the new expertise that needs to be updated and accumulated by the next generations? I'm curious: from a data scientist's perspective, someone who's actually creating these systems, what's your point of view on this?

Ahmad Fattahi: It is not an easy question, in my opinion, because what I see are two opposing forces. The immediate reaction to the existence of all of these capable AI systems is — many of us have heard this idea of the inverse-pyramid structure for teams and organizations. In the old world you had a small group of highly experienced experts at the top, and as you got more junior, where the need for expertise was lower, the pyramid got wider. So you had more junior people: the true pyramid. Now with AI, the immediate reaction by a lot of decision makers and organizational designers was that all of those more junior tasks and needs are now capably handled by AI; therefore, to go back to my earlier point, we need more expert people at the top, and we don't need the junior tasks as much. So, inverse pyramid. That's force one: the economic benefits or incentives of going inverse pyramid, because you're going to reap the benefits.

Ahmad Fattahi: But then the opposing force is: okay, now let's imagine the year 2032. Who's going to be at the top of that inverse pyramid? The people who historically, over the past 5, 10, 15 years, were at the bottom of the pyramid — but we are hollowing out the bottom of the pyramid. So as those people at the top either move out or retire or whatever, how do we replenish the top of the pyramid? Traditionally there was a wide bottom of the pyramid that we are now hollowing out. So in the interest of longevity of the business and the organization, we still need to beef up the bottom of the pyramid. Some people argue that we need an hourglass-shaped environment, where we still bring in more junior people and expose them to the actual operation of the business, so they learn by doing the traditional way, so that they are ready in five to ten years to act as the top of the pyramid, the human experts, while AI still delivers the benefits in the middle of the organization. These are opposing forces. There's always — especially if you're reporting quarter-to-quarter to the street — this temptation that, okay, as long as we can manage now, we'll deal with the expertise three years from now. So the jury is still out. But these are the ideas that are out there in terms of the impact on the shape of organizations.

Victoria Mensch: Yeah. And I think it also brings up the question — and you already mentioned it a few times in this conversation — of that balance between the efficiency and productivity gains that we can harvest right now from technology like AI, and that bigger transformational drive, the shift it can really offer from a business point of view. Where do you personally draw that line? How do you actually balance that push for productivity and efficiency gains versus really cultivating that transformative force that AI can be?

Ahmad Fattahi: So, if I may, I'd like to answer it in a slightly different way. There are actually opportunities today where AI can be both a force for more efficiency — which is typically the first thing people think about when they think about AI: instead of spending three days crafting a marketing message, we can now do the same thing in two hours; that's a clear, obvious efficiency — but also, in many cases, as we speak, there are many opportunities to improve the effectiveness, the quality of the work, with AI at the same time. So it becomes a no-brainer, or very hard to resist as a decision maker. To continue that example of the marketing message: imagine that your AI can also tap into many more sources of insight about your audience, the recipient of the marketing message, compared to the human marketer. Not only does it do the job faster, it also crafts a better, more effective, more personalized message to your audience, which is one of the main factors for any marketing activity.

Ahmad Fattahi: In my observation, there are those opportunities which are no-brainers. You don't even have to make a sacrifice or a compromise; you're not saying, 'okay, with a little bit of reduced quality we can do more, so more efficient — therefore let's make a trade-off.' In many cases you don't even need to do that: you actually gain both effectiveness and efficiency. Those are no-brainer tasks. The question of expertise in the future, or other factors, becomes much harder to make a dent in. The decision that 'hey, we can do something better and faster' starts to break into that whole good old consultant's triangle: out of the quality, the price, and the speed of any project, you cannot have more than two — at least one needs to suffer. Now with AI, you start to think that maybe there is a way out of this dilemma of having to drop at least one of the three. So, going back to your question, instead of trying to make that tough trade-off, I would start from these opportunities that I would argue actually abound. There are many opportunities in any organization where you can use AI to do things better and faster and cheaper at the same time.

Victoria Mensch: Well, that's an interesting point of view. That would be fantastic, right? The ideal environment where we can do all of this at the same time — cheaper and better. But a lot of businesses really struggle, and I think it's also a matter of mindset, right? How do I actually transform what's already working? We have our status quo, we have the processes that work, we have the business model that works. Now this new technology comes in. Usually the temptation is really to improve what's already working, but the challenge, from the innovation point of view, is really to rethink the process. Going back to your example of marketing messages: what if there is a world where we don't need to create marketing messages anymore? How do we account for that when we're making a decision right now about what to implement and how to implement from the AI point of view?

Ahmad Fattahi: Sure. I said everything I said. Getting there — the how — is a different question. I don't claim that it's always easy, and I'm not claiming that it's risk-free. All I'm claiming, fairly strongly, is that those opportunities exist. But getting to that point still needs proper thinking by, ironically, humans: very capable, smart, expert humans who have done these transformations. And these are often pretty significant transformations for any organization. There are many pitfalls, many ways to get it wrong. There are many examples of great technological solutions that actually nobody used because they were done wrong, or because it harmed human morale, the organizational morale. There are examples — I think I can say this was in the public news — like Uber, the company. They're at the cutting edge of many transformations; they transformed a century-old industry themselves. To leverage AI in their organization, they incentivized their workforce to report back how many tokens they used, because that was the proxy measure of how much they leveraged AI. But it backfired, because people started to take the proxy measure as the goal. That's human nature.

Ahmad Fattahi: They started to burn tokens, sometimes intentionally, just to go up the ladder of the leaderboards and show up as a bigger user of AI in the organization. So it backfired, and they changed their program. The point being: there are many ways to get it wrong. But those opportunities definitely exist. And a bigger point that you raised is where the opportunity is actually rethinking the process altogether with AI. That's actually even harder and, at the same time, a higher-reward opportunity. The example I read a while ago — I think it's a brilliant example — is that in the beginning of the industrial revolution, when the steam engine was invented, all these textile factories were built next to rivers as one of the only feasible sources of energy: the running water to turn the wheel and make the factory work. When the steam engine was invented, for the first 10 or 20 years, it wasn't easy to rethink, to get out of this jail of 'textile factories are meant to be next to rivers,' and nobody rethought them for 10 or 20 years.

Ahmad Fattahi: So they just ripped off the water wheel and replaced it with a steam engine. They still got some efficiency gains, but it took 10 or 20 years to start thinking, 'wait a minute, why are we even building these big factories next to rivers? We don't need to; we can optimize by being close to railroads or close to the population centers where more workers can help us.' That's a perfect example of what you're referring to. The process efficiency gain was the immediate thought, but rethinking the whole process took a while. And the parallel may apply here in the age of AI.

Victoria Mensch: Well, thank you, Ahmad, for this perspective. I think what we definitely uncovered today is that, really, not a lot of people have answers right now, right? There's no right or wrong way of doing things; we're learning about the impact of AI as we go. And some of those shifts that we're talking about, I would say, might be heavier on the people side than the technology side. So I'm really curious: what does that mean for you, to lead your team day-to-day, when you yourself don't have all the answers?

Ahmad Fattahi: It's not an easy topic. Sometimes it's easy, not always. As a leader, the news itself is not easy. In this case, the bigger question is the ambiguity, because nobody has the answers. Almost everybody expects a massive change that's going to last for at least a few years, with potentially very highly consequential impacts on people's careers and jobs and on the way people run organizations. That's the general expectation in many minds — not every mind, but many minds. But on the other hand, there are very few confident answers. As you and I were chatting for the past few minutes, there are sometimes very big differences in assessment by experts in the industry and in the AI field about what will happen — whether the future of AI is AGI, whether AGI is even a possibility or not. At the highest levels of the echelon of the AI field, the biggest names sometimes express different opinions about that. So that makes the process of leading a team, leading humans, even more challenging, because there is no clear, confident path forward.

Ahmad Fattahi: So the best is the good old: pay attention to the humans, have conversations with them. Even the fact of acknowledging that 'these parts we do know, these other parts we don't, and we are all in the same boat' goes a long way. I'm now switching to the more human aspect of organizations. It's very different to say nothing, or, in contrast, to acknowledge that there are unanswered questions and that we're not hiding anything — the organization is not hiding anything. There's no nefarious backroom, dark room, smoke-filled decision core deciding, 'oh, we're going to cut this percentage of the organization and replace it with AI.'

Ahmad Fattahi: Building trust — the good old best practices of people management and people skills — absolutely applies here, arguably way more than before, because you need those people. Still; the jury is still out on how much of the work is going to be replaced literally by AI. There are still opinion pieces that I read that say, yes, there are these fancy AI models, but the humans are doing different things and are actually working even harder to manage and orchestrate those AI agents. So it's far from clear. The best you can do is to stay close to your employees, to your team, up and down the broadest team. Gather insight — to your point, for example, of reimagining the whole process. Many of these great ideas actually happen when this Brownian motion happens: people from different parts of the organization talk to each other, sometimes outside the box. By nature, these big transformational changes require thinking outside the box. So when you increase the chance of running into vastly different ideas and viewpoints, you increase your chances of leveraging AI to rethink the whole process of your organization while keeping people motivated and bringing them along.

Victoria Mensch: So just to wrap things up, there's obviously uncertainty, right? And I think the whole environment is getting more and more uncertain, and that's making your decision process much more challenging from a leadership and executive perspective. Is there anything you would like to leave the leaders in our audience with? One principle that helps you make better decisions in situations of uncertainty?

Ahmad Fattahi: Sure. The one immediate thing I can think of is that not making a decision is a decision in and of itself. Sometimes we assume, 'oh, I'm just going to punt it, I'm going to wait, I'm going to gather more information.' I'm not by any means advocating for rushed or incomplete decisions. But what I am saying is that we cannot assume that just waiting it out is going to protect us from making a potentially incomplete or imperfect decision. In many cases, not making a decision — not moving — is even more harmful to the organization and to the people themselves. It just increases the pressure on decision makers to move faster. The move could be an intentional wait, but it has to be intentional. It cannot be just, 'hey, let's wait it out because we don't have enough information.'

Ahmad Fattahi: So bottom line, it's harder. Acknowledging the uncertainties and the level of uncertainty with the team — transparency — hopefully creates a little bit of wiggle room for decision makers to bring the team along and say, 'hey, we have to make a decision; not making a decision is a decision in itself, and the risk of doing so may be higher; therefore, I need to make a decision now to bring in this system or apply this AI or do whatever' — or sometimes say no to an AI; that can also be a decision. We acknowledge that there is a chance that this decision is wrong or imperfect, but considering everything we know now, that's the least imperfect decision. Communicating that, translating that to your team up and down the chain, is very helpful.

Victoria Mensch: Thank you so much, Ahmad, for this conversation. To wrap up: I think we talked about so many very important issues. One of those that I really enjoyed is the balance of risk and reward in implementing AI in a variety of industries, and how regulated industries in that sense are different from those that are not subject to heavy regulations. We talked about building layers to mitigate risks within those systems: on a data level, on a level of instructions, and human in the loop. We talked about the importance of human expertise when designing these systems within the organization, and really those massive changes that AI brings to organizations. And I really loved your last point, that inactivity and waiting it out is really not an option, and sometimes you have to compromise on going ahead with the least imperfect decision rather than indecision. Thank you so much for being here. And for our listeners, if this conversation was useful, follow the page on our LinkedIn so you can catch the next one, and drop a comment below and we'll make sure that Ahmad is there to answer your question.

Ahmad Fattahi: Thank you for having me, and it was a great conversation.

Victoria Mensch: Thank you.