Can We Trust AI in the Physical World? Responsible AI & Scaling Beyond the Pilot

Victoria Mensch: Welcome to Decisions at the Frontier, brought to you by Silicon Valley Executive Academy. I'm the CEO of the academy, where we help senior leaders tap into the Silicon Valley innovation playbook. Today our guest ran global go-to-market strategy for AI at Amazon Web Services. He currently leads FinTech4Good, a global network scaling AI adoption in financial services in emerging markets, and he leads AI 2030, which works on making AI governance operational rather than theoretical. That combination — working on the infrastructure, building the ecosystem, and stress-testing what it produces — is really rare. What I find most valuable is that he's not just talking about AI transformation from the sidelines. He has made his bets, watched some of them fail and some of them work, and built the frameworks that came out of that. Xiaochen, welcome to Decisions at the Frontier.

Xiaochen Zhang: Thank you for the invitation. I'm really looking forward to our conversation.

Victoria Mensch: What I find interesting in your background, Xiaochen, is that you've worked across so many different industries, functions and regions. Most leaders try to specialize. What did you gain by having such a diverse experience?

Xiaochen Zhang: I think it's like how we train a large language model. You bring different experiences into yourself, and naturally you're exposed to more situations, more problems, more diversity, more solutions, and more failures and successes. All of that becomes part of who I am, and I can always bring a piece I learned here or there to exactly what I'm doing.

Victoria Mensch: Great. Can you tell us a little more about what you're doing now?

Xiaochen Zhang: Absolutely. My main focus is trying to mainstream responsible AI by 2030. That is my single largest goal for now, and the reason is very simple: AI is a general-purpose technology, which means much of its potential impact will become more present as time goes on. So I bring all my expertise, my exposure and my access to solving this problem — bringing the right people to the table, discussing the right topics, designing the right frameworks and action options, and helping people take action so that those accumulated actions lead us to a better future.

Victoria Mensch: The topics you're talking about are really of global importance. I absolutely agree that the technology itself is neutral, but as a society we're at a crossroads. There's a choice to be made — whether we turn left or right — and I'm very happy to see you facilitating that conversation at that level. Now let's step down a little from that level. What we often see is a chase for productivity within organizations: how can we do things cheaper and faster? But there's an angle that's probably more important with a technology like artificial intelligence, which is its transformational power. Is this something you've experienced working with different organizations? What would you advise companies to do, and what framework puts the right priorities in place?

Xiaochen Zhang: What we've observed in the past few years is that everyone asks the same question and puts capital and human resources into the same thing: what can we build? AI can make everyone a builder and build systems that are very capable. That is the story of the past five years. But if CEOs are still thinking that way, in the next era — the next five years — the most important question will not be what you can build or what capability you can bring. It will be whether you can build systems that people trust. That will completely change the priorities for leaders, for builders and for customers. Everyone has already seen the capability of AI. At the enterprise level, losing a single court case can mean $1.5 billion. But if people lose their trust in a company, especially a rising company, how can you put a dollar amount behind that? So trust becomes really important.

Xiaochen Zhang: Trust itself can be divided into many layers: the data layer, the model layer, the compute layer, how you manage your own employees, how you treat your customers, and how AI decision-making is managed by humans — whether humans are effective when they are the designers, or whether AI is developed in a way that alienates human processes, human relationships and human needs. If we don't want that to happen, humans have to be at the table from inception to design to execution, and we need to bring trust into every layer. That is very critical, but few are doing it well, and few even know how to do it.

Victoria Mensch: There's definitely a crisis of trust. I often say we're really at the point of a crisis, where you don't know what to trust anymore — even on an individual, everyday consumer level. You get a text and have no idea whether it's from a real person or whether it's artificially manufactured and someone is trying to scam you. That starts at this level and scales up. You're talking about all these layers within the organization. Employees don't trust employers anymore: are you using me to train the models so they can substitute me going forward? There's so much around trust. But an executive in a company might say: yes, these are big, fundamental questions, but what can I do about it? I have my area of expertise, my agenda, my company. What would you tell that person?

Xiaochen Zhang: This is the type of conversation we have a lot. Just two or three days ago I gave a talk at an event with 200-some executives, and one thing I shared is that AI is moving from digital to physical. That's one of the big trends they need to be aware of. In the digital world, the worst that can happen is that someone loses their digital identity. Their digital self is impacted, and there can be financial and other harm, but that person is not physically hurt. When we move from digital to physical, AI is going to live in your bedroom, in the bank, in the hospital, in the surgery room. When AI does something wrong there, someone is going to get hurt. So for many leaders, based on what services you provide to your customers, you need to understand that when human life is in your hands, this is not abstract. It's not something at a higher level; it will impact your daily business. If your product hurts people, you'll be responsible both legally and financially, and you may lose your customers' trust. So that's number one: as a leader you need to understand this is very concrete to your product design.

Xiaochen Zhang: Number two: once you have the mindset that, moving from digital to physical, we need to prioritize trust, the question is how I, as a leader, bring that trust into my process. You have to back that story and raise the flag. Saying “you have my sponsorship” is not enough — you have to own it as a leader. If you don't own it, someone else won't have the mandate. At the end of the day, if that person is fundamentally transforming your product, your customer experience, your workflow or your employee experience, and you still have the final say and can simply decide you won't sponsor it anymore, that's too easy, and that person or team will fail. So whoever is number one in your organization needs to own the process that makes trustworthy AI the number one focus when you deliver services to your customers.

Xiaochen Zhang: Number three is resources. I talk to many CXOs whose CEOs are not fully convinced. When I was at HumanX, I listened to Fei-Fei Li. She was asked how she felt about raising over a billion dollars, and she said she wants to bring AI out of the darkness: the problem is big enough that she needs the best engineers and large-scale computing to make world models possible. That is resources — making AI trustworthy needs resources, so leaders need to give the right teams resources and a mandate. The last piece is to engage your customers and bring them to the table. I've been part of several AI customer advisory boards at large companies, and for me that's a very strong signal. A lot of larger companies have started to think that way. Of course the scale is still very small — they curate a list of people, the conversation is limited to that small group's experience, and even that group's impact on the product process is limited. But at least we see institutional design that brings the customer's voice into the design, development and building process. Those are very concrete steps the CEO can take, or the board can ask the CEO to take, if they think trustworthiness is important in the next phase of AI competition.

Xiaochen Zhang: I've developed two toolkits in the past to guide organizations in adopting AI. Number one is value alignment. If the AI doesn't align with the values of the company, it cannot go far. It can stay there looking very exciting, but without strong value alignment, the AI pilot will be killed easily. Number two is the real customer problem and business-unit buy-in. I think modern organizations have a serious problem here. To simplify the organization or improve efficiency, we establish boundaries between departments, and everyone owns a piece of the value of the whole customer journey. That makes scaling innovation or AI pilots very difficult. You have an innovation team working with external innovators to build paid proofs of concept, and if the business-unit leader isn't fully convinced, there's no business unit to take it over and commercialize or productize it. That kills the product immediately.

Xiaochen Zhang: Of course, some pilots are purely an internal learning process. Not all pilots need to scale. Many pilots in a large company exist so leaders and the company can learn. Failure is expected; they want to learn from the failure and the process so leaders are aware of the opportunities. Those are, by design, not meant to scale. Then, if business-unit leaders are convinced but start without the customer's pain points in mind, it becomes just a business decision. If the business unit wants to translate it into, for example, an insurance policy or a financial instrument, and there's no demand from the customer, it will not scale. Every product is sensitive to time: if it's already out of the lab and then paused for a few months, that alone can kill it. So it matters whether you brought your regulator along with you, so that when the product is ready, your regulator is also ready to help you, or at least give you the green light. The last piece is internal process. These proofs of concept or pilots can bring a lot of new workflows, new customer experiences, new interactions and new expertise. If your team isn't ready, it will be very difficult for any pilot to succeed, because you don't have the team to run it.

Xiaochen Zhang: So, going back to your question, the number one answer is that you need to know who you are. Whether you're a startup, a growth-stage company or an enterprise, you face very different challenges in scaling an AI solution. If you're an enterprise, everything is about internal process and operational excellence. At the growth stage, it's really whether your customers have bought in and want it — you can test it with a small group, but if your customers don't want it, you cannot grow. For a small startup, it's whether you can get the capital and real customer validation through your scaling process. Again, depending on who you are, success looks very different.

Victoria Mensch: So the challenges look different depending on who you are. It could be an organizational decision — whether the organization is actually set up to scale a pilot. It could be a leadership decision — whether I want to prioritize the scaling effort. Or it could be a technology decision, depending on where the company is in its growth cycle, its industry and its ecosystem, as in heavily regulated industries. But in general, is there a common thread you see — efforts that have a better chance of moving to scale than others?

Xiaochen Zhang: The ones that tend to be more successful, number one, include the right stakeholders at the table from the design phase. That changes the whole game. A lot of the time, a product team doesn't really want to engage legal or compliance from the beginning, because they don't understand enough about the product. Then you need to explain it, and after you explain, they tell you all the things you cannot do — which are probably the things you really want to do — and that can discourage the whole team. We already know that bringing others to the table makes the process longer and more time-consuming, but at this phase, starting with everyone's participation, learning and buy-in is critical if you're thinking long term and about scaling, not just a proof of concept. If I'm leading a unit and want to learn something, I can run a pilot without engaging others, but that's only for the sake of learning. If you really want to scale it, you need to engage stakeholders from the beginning.

Xiaochen Zhang: Number two: you don't only need to engage stakeholders in the beginning, you also need to know who the most critical people are who have to be there. For a specific product, you always see missing expertise, and that missing expertise becomes a major problem later. One AI story: a government tried to use AI to allocate subsidies to people with less healthcare coverage, and the AI, trained on the data, actually gave more support to those who already had a lot more services than those who didn't. The system was designed in a way that was completely against reality. If you have people from those backgrounds at the table, they will naturally ask exactly that question. Women in AI is also a very important topic for us. If you deliver AI products for a gender that is missing from the engineering team, or you're building a product to serve people in remote areas without access to a computer, bring those people to the table. I can guarantee you'll learn a lot, and the product will be a lot more feasible.

Xiaochen Zhang: The third is a very clear return defined in the product strategy. A lot of the time with pilots, people say: give me a small amount of money, this is all I need, I just want to get it done. If you as a leader have that mentality and only ask for that amount of resources, you'll end the project with exactly the resources you were given. When you really want to scale, you need a five-year plan and the right resources. When you ask for leadership approval, you need to budget for at least the time you need to scale. Otherwise you don't have enough people to test it, enough people to go to market with, or enough legal experts to advise you in certain markets. Finally the person leading the project says, “I cannot do this — I only have 24 hours, my team is working very hard, we just can't.” Not asking for the resources tends to make a lot of products fail.

Victoria Mensch: It's all to say these are complex decisions, in larger organizations and small organizations alike.

Xiaochen Zhang: It's not only complex in decision-making — there's also a trade-off, and a lot of it goes back to speed. Everyone wants to succeed quickly in their job, to prove that this is my result, and…