Last year at The AI Summit New York 2022, seven experts engaged in a discussion about the future of AI, and a singular theme emerged: the influential role of collective intelligence. 

This dynamic concept, which amalgamates human insight with AI capabilities, is reshaping industries on a global scale. 

During this panel, now made available to watch on-demand, industry leaders shared how to unlock the full potential of collective intelligence and make it accessible to all. 

Thanks to our panelists:  

• Maryam Farooq, Founder & Director, New York Artificial Intelligence (NYAI) 

• Abhishek Kodi, Principal Business Architect (AI/ML), BNY Mellon 

• Theodoros Zanos, Head, Division of Health AI, Northwell Health 

• Rohan Doctor, Founder & CEO, Louisa.AI 

• Prof Carlos De Oliveira, Director, Senior Principal, BNY Mellon 

• Cynthia Freeman, Research Scientist, Verint 

• Kriti Kohli, Senior Manager, Applied Machine Learning, Shopify 

The insights from these industry experts highlight the importance of creating user-friendly interfaces, managing risk, and maintaining trust in AI systems. The power to shape the future of AI lies not just with developers but with designers and the broader community.  

Explore our agenda for The AI Summit New York 2023 and learn how you can be part of this transformative journey.  

 

The Transcripts:

00:00 Maryam 

Hello, everyone, welcome panelists, please get comfortable. We'll be here for 45 minutes. All right, as Joe said, I'm Mariam Farooq. I am the Co-Founder and COO of a startup called Aggregate Intellect. We are building a collective intelligence platform to help developers make product decisions faster and better. We use a combination of NLP and graph analysis to generate interactive mind maps of your team's collective knowledge. So, if you want to learn more, come find me after the talk, or, you know, reach out to us at AI data science. 

Okay, so I'm very excited to moderate today's panel, the prep call for this panel went by really quickly, I think we're all very passionate about the topic of, you know, the, how the sum of the parts can be greater than the whole. So we're gonna kick things off with just a few notes. Also, I think we can all agree. so we have great panel today, they have all worked in a variety of you know, academia and industry, some have worked in both. And I think we can all agree that you know, regardless of what industry that you are working with, the idea of collective intelligence is very compelling. So, there's many ways to define collective intelligence, from a psychology perspective, an engineering perspective, we're kind of going to be coming from it from the Human Computer Interaction side, and how AI can help kind of spur innovation all over, I want to quickly share my favorite definition of collective intelligence, which is actually from 1994. From Smith, a group of human beings carrying out a task as if the group itself were an incoherent, intelligent organism working with one mind, rather than a collective of independent agents. And of course, rapid advances in technology like AI, there are many more opportunities. To progress this forward, we have seen things like Citizen Science, crowdsourcing things, whether it's passive or active or even more directed. But today, we're going to be talking more generally about how you know the challenges of harnessing collective intelligence within the various industries that our panelists are working in. So with that, I'm gonna pass the mic to everyone and we'll do a quick introduction. Theo, would you like to start? 

02:52 Theo 

Sure, can you hear me okay? Hello, everyone. My name is Theodoros Zanos, I’m the Associate Professor at the Feinstein Institute for Medical Research at Northwell Health and the Head of the neural and data science lab, where one of the world's largest healthcare system in New York with 23 different hospitals, more than 700 patient clinics, and my team is developing AI solutions for the frontline workers or clinicians, or nurses and eventually trying to improve the care that we provide to our patients. 

03:32 Abhishek 

Hey all, This is Abhishek Kodi,, I'm a Principal Business Architect BNY Mellon, the oldest bank, almost 238 years now, since founding, and BNY essentially lead a center of excellence team for data science, mainly for a couple of LoBs, clearance and collateral management and credit services. As part of that I build, I'm hands on data scientist as well. So I strategize some of the best ways to scale up AI and ML solutions across our LoB across operational teams and business teams. And I'm very glad to be part of such a good cohort of speakers. 

04:13 Cynthia 

Hello, okay, perfect. It's on. Hi, I'm Cynthia Freeman. I'm a Research Scientist at Verint. And what makes me unique is we have a very easy to use orchestration platform for helping us obtain executable insights with customer data from a variety of channels. So as a research scientist, it's my job to look at all this data and see if we can improve our products like our chatbots, or with time series anomaly detection to make it easier for our customers. I'm very excited to be here today. 

04:49 Kriti 

Hi, everyone. Kriti Kohli, I'm a Senior Manager of Data Science and Machine Learning at Shopify. Shopify is one of the largest ecommerce platforms out there and what my team does is we build out AI ML solutions for our marketing suite. So that includes things like Shopify email, or messaging. And it allows merchants to directly interact with their buyers. 

05:20 Rohan 

Hi, everyone, I'm Rohan doctor. It's my 17th year at Goldman Sachs. And, you know, it's quite serendipitous that I'm here on stage with, you know, collective intelligence, expertise, because it really all happened because one thing led to another and I closed a really large trade when I was running bank solutions in Asia. And, I got the chance to do a victory lap and see the c-suite in New York. And they said, How did you do that trade? And how do we repeat it? And you know, the honest answer was, it was a little bit of luck. I met the right person at the right time, they mentioned something, someone else mentioned something and it happened. And they're like, Well, how do we repeat that? And as like, you've got to almost systematize serendipity. And so that's the platform we've built called Louisa within Goldman. And so now I run an AI and collective intelligence team at Goldman Sachs, which, which basically makes sure that, you know, chance meetings happen more and more often using data rather than just luck. 

06:34 Carlos 

Hi, my name is Carlos de Oliveira. I have about 27 years work experience on both the investment industry and academia. I work with massive datasets. So currently, I work for the Bank of New York, which is the largest custodian in the world. I am the Head Scientist there. So my research is mainly on macroeconomics, using as I said, these gigantic data sets as well as the projects that I deliver for the bank. 

07:13 Maryam 

Awesome. Thank you to our panelists for the intros, systematizing serendipity is something that I really like and will probably still so thank you, Rohan. I, so I'm on this panel. We have healthcare, we have banking, we have some consumer tech, I'm curious, you know, what does collective intelligence mean for you and your industry? Specifically? Theo, why don't we start with you? 

07:38 Theo 

Sure. I think when it comes to deploying and using AI tools in healthcare, we essentially use collective intelligence, even if we don't say we do, mostly because these tools, at least in the moment, and for the foreseeable future, will not be able to work by themselves. So there is no AI tool right now that's deployed in any type of hospital that will automatically give a treatment to a specific patient or diagnose somebody with a disease without a doctor signing off on it. So inherently, the human, the loop component of collective intelligence is omnipresent in healthcare applications. But beyond that, the idea of integrating data sources from various different places, whether that's imaging, radiology, medical notes, labs, and vitals and demographics of patients, or medical, or extracted information from their past medical history, all of these land themselves into collective intelligence framework, mainly because they're coming from completely different places within the organization. And they need to be integrated in a smart way. And finally, I think collective intelligence is extremely important in healthcare, because it will be used by the frontline workers or clinicians or doctors or nurses. And they need to be part of both building these models from the get-go. Because they will suggest based on their experience, what is the most important problem to be solved, but also where in their workflow this tool can be deployed? What type of data will it use? And also they could suggest based on their medical knowledge and other published research in the in the specific topic, what should we be looking at when we're building these tools, whether we should be looking at specific labs that are relevant? So there is a huge component of collective intelligence when it comes to medical AI that I think right now because we're not calling it that, it might be lacking a bit of structure. And I think that's where these discussions and in general the field can provide a lot of value. 

10:11 Maryam 

Yes, absolutely. Especially in an industry, as you said, that the guiding principle is do no harm human, the loop will absolutely be critical. But I think the human in the loop component will be important for every collective intelligence platform. Abhishek, what are your thoughts? 

10:28 Abhishek 

BNY in the financial services industry? Right, so expertise is key. Moving regulations are key. So in order to really standardize, scale, and mitigate risks across all our services, front office, back office, and also regulatory compliance, AI has become a central tool to standardize scale, and apply these in an effective way. I think the easiest example I can give you is fraud detection scenarios, there will be experts who would go through fraud, fraudulent transactions, identify the rules, or patterns that could be leveraged. And that process still exists, but a layer of AI on top of it, which collects expertise, opinions, facts, and a combination of an aggregate of all these into an actionable process, through an AI built is really the power and really, in a sense of subset of AI. The collective intelligence really dominantly overlaps with the supervised learning use cases, the classification kind of tasks most people do are in just another form of expressing collective intelligence trying to capture from the experts and ensure that that's democratically spread across our processes in order to leverage it for business processes.  

11:48 Maryam 

Right, I love that the democratization piece kind of fits in with the human in the loop piece within that. I would love to actually hear Carlos his thoughts after that, because you're both in banking and at BNY. 

11:59 Carlos 

So yes, I think that collective intelligence on the investment industry is all about matching. The very rational mindset of AI, very statistical based mindset of AI, with the very irrational decision-making process from investors, that is very complicated match. We don't get there in many situations. Just to give you an example, I think I was one of the first Wall Street guys to publish an article stating that the inflation was not transitory. That was back in March this year, when the Federal Reserve was basically saying that inflation was transitory. So what? Can we make a ton of money out of that? And the answer is no, we cannot because the markets in March, they were not buying the idea that inflation was not transitory. But that's the way I defined collective intelligence. And it's a very complicated match for the investment industry. 

13:10 Maryam 

That's interesting. Why don't we wrap out the regulated industries with you, Rohan? 

13:18 Rohan 

I actually like that thought, because the market is the ultimate collective intelligence in many ways. Yeah, for me, it can mean collective in so many different ways. For me, it was all about human collective intelligence. But enabling that requires the best of man and the best of machine. And so for us, it's like, we've got 50,000 people at Goldman Sachs, across, you know, different divisions, regions, expertise, traders, bankers, legal, etc. And so, you know, just being able to say, who had the firm can help with anything, and being able to leverage that person's expertise, to help with whatever situation comes across, you're gonna come across either on a news article or a client inquiry, or an idea that pops into your head in the shower, like whatever it is, if you can just say, I wonder if someone at my firm knows, I can leverage someone at my firm about, you know, and figure that out. That for us is collective intelligence, and how it started. And we've built a platform that enables that at the firm, and we're spinning out in Q1 And so we'll be able to enable it for many organizations. And so for me, it's really the, you know, the whole of management committee at Goldman Sachs can't know as much as what you know the context in a single iPhone. So that's like machine can beat us at that, but a five year old has more general intelligence than the best computers and all out there. So it's really taking the best of both and marrying it. 

15:11 Maryam 

Yes, absolutely. I think it's important for us to, you know, I think we're coming from the human intelligence perspective, we, you know, technical intelligence can exist in a variety of sources, whether it's a scientific research paper GitHub repository, there might even be conversations on slack with your colleague collecting the human piece. So you know, how are organizations currently seeing all of the information out there and creating a unified view of it so they can make better decisions in general? I'd love to hear from the consumer and the insights, tech side, Kriti? 

15:49 Kriti 

Yeah, I think, from Shopify perspective, one of the main things that we want to do as a company is make commerce better for everyone. Right. So that's part of the mission. When you think about the collective intelligence that we're looking at. We have a lot of different scales of merchants, there's small merchants, there's very large merchants. And they're competing against retailers. So in a way, what we want to do is bring these AI ML technologies have the collective intelligence across millions of these merchants, and empower them with having access to the same technologies that large retailers have. So if you think about, like the small mom and pop shops that are on that have built out their own online business, a lot of them are not going to have access to advanced things like ChatGPT. But what Shopify can do here is taking you know, here's like, what we think is the typical sort of message that you might get from a buyer coming in visiting your storefront, we can take that information and provide merchants with a way of being able to answer questions faster. So we also know that, you know, if they are able to answer questions faster than they typically lead to higher conversions, buyers will actually buy something on their website. So we're providing them with this technology that isn't really accessible on a on a day-to-day basis for these merchants, these small merchants, that retailers have like access to large compute resources, teams of data scientists. So in a way, we're kind of pooling the information that we are seeing as patterns across small businesses. And that's another way of providing a collective intelligence. 

17:52 Maryam 

Definitely. And last but not least, Cynthia. 

17:56 Cynthia 

So when I think of collective intelligence, I think of a group of people who come up with cool idea. And it's way better than if just one person works on it, right. So if you have the AI, you're adding some scaling, but you still need to include that human component, because otherwise, they might make some not very ethical decisions, or it just gets it all wrong. So fairness of customer engagement company, and one of the things that we work on is anomaly detection in time series data. So an anomaly in one time series domain data domain may not be an anomaly in a different domain, you can't really have just one definition for anomaly, right? So you may just grab some anomaly detection method, you find some research paper and hope that it works. But it doesn't really work. You need to have that human in the loop component, where they can kind of look at the anomalies that are predicted and say, You know what, I agree or disagree with you, right? And then the machine takes that data and goes, Okay, then I'm going to look for things that are really similar to the instances where people say, Yeah, I actually agree with you, I think that's an anomaly. Otherwise, your machine is just in me throwing about throwing out a bunch of like, false positives, and people are gonna ignore your system, and you're gonna suffer from alarm fatigue. I mean, another idea is another one for products is trace. So we have a lot of chat bots out there, right, and the customers in the customer space, but sometimes they misinterpret what the customer wants. So there's an intent problem. So if you can just focus with the AI, what kind of intents are problematic, and then give that to humans to tag that makes everybody happy, right? You got the AI partner helps you scale, but then you got the human component that makes the AI more human. So you need to have that kind of merge between the two. That's what I think of when I think of collective intelligence. 

19:47 Maryam 

Awesome, thank you. I think it's really interesting that within the variety of industries that we touched the recurring themes were human, the loop, you know, any model that is going to have some impacts on a consumer or a human or a patient is going to be require that humanistic element. And then we also touched on things like the, the regulatory constraints and compute restraint, constraints rather. And also democratization I think I heard in there as well. Interesting, I would love to go back to Theo, you touched on it before, saying that you don't really have any collective intelligence platforms within healthcare right now. But you mentioned that there's something called Learning Health Systems, I'd love to know a little bit more about that, and kind of what is left to be desired, within the ideal platform you'd be looking for? 

20:42 Theo 

Yeah, that concept of Learning Health System is, I guess, not very new. But it's not also very widespread. And when we look at the the idea of Learning Health Systems is this circle where you have certain scientists or physicians gain knowledge, usually, and mostly from data collected, they that knowledge is used through algorithms or other change management processes, to actually develop new solutions or new tools, they are used, they are monitored, followed, and then that circle just keeps on going. Because these tools now produce even more information that can be used in a circular manner. So that to me, almost sounds like a rebranding or kind of like, it's describing a similar concept as collective intelligence. But I think collectively, this is kind of like an umbrella that kind of extends a lot be a lot more and beyond these, these narrow definitions, because I could think of collective intelligence special in the healthcare domain as a component of it could be a tool that could aggregate all the published, let's say, research and literature in a specific disease, and summarize that and provide that as inputs to or information to either a doctor that is treating a patient, or to a modeler that wants to create a diagnostic or prognostic model and point them to the right direction, which data you should be looking at how you should, you know, structure your model, so that it's, it's accurate, and it's also usable. So there are a lot of things that I, you know, by talking and hearing others come to mind on how this collective intelligence could be broadened from the narrow, let's say, scope of like just a learning health system, or just an another AI tool. And I really believe that it will improve the algorithms, but it will also improve the adoption of these algorithms by the stakeholders that which are the doctors, our clients are the doctors are not the patients that because we always have that, you know, human in the loop that sees the prediction of the algorithm and says, as a, you know, like, is this really an anomaly as you just said, or is it just a false positive? Right? So in that case, I think, a framework that encompasses all these different possibilities, as far as I know, is not very well known or adopted, but I think it would be useful to develop. 

23:30 Maryam 

Yeah, it's interesting, that it feels like there are a bunch of, you know, point solutions that people are, you know, kind of creating their own ensemble frameworks to achieve something. But, you know, ideally, we'd have some sort of system design at the collective intelligence level, to make something actually actionable at the end of it. I feel like I've seen a lot of people using, you know, like note taking apps notion or Confluence you probably use in your company. There are some second brain frameworks out there, like Rome research, or obsidian that's more on the personal individual level rather than the collective level. And then, of course, there are a bunch of knowledge hubs and model hubs like GitHub or hugging face. But right, being able to put those all together and give you the knowledge in time when you're looking for it is kind of the next step to be desired. Rohan. I'm curious, because you're coming from this question, really, from the human intelligence level. Tell me more about Luisa and how they're thinking through that platform. 

24:35 Rohan 

Yeah, I think I think your platform does it very well for the engineering community. Is that right? Yes. And so you know, even at a place like Goldman Sachs, we have you know, 10,000 clients facing people 10,000 engineers, and then 20,000, you know, HR, Ops, legal, etc, etc. So, you know, we've got something I don't think it's is probably as developed as yours, but we'd be interested in thinking about it. But basically, it's, you know, since COVID, many firms have had this battle of you know, is it all, you know, I was interesting, Shopify is completely remote, right. And even though lots of my colleagues 20,000, or they're just remote, and Goldman Sachs, we've had the opposite. We've had our CEO wanting us to return to the office. And I personally love meeting people and interacting with people live vs zoom. But it's crazy that in today's day and age, where we have AI summits, one person has to be thirsty on the floor, and another person's thirsty, and they come to a water cooler, and they have a water cooler conversation. And that's how knowledge is transferred. And that's essentially, you know, I'm kind of laboring the point, but you know, whether you're meeting at the water cooler, or an elevator or on a pitch to a customer, that's where a lot of discussion and innovation and sparks and knowledge transfer happens. And it's great when it does, but, you know, it needs to be more systematic, right. And so, what Louisa does is every morning before all 50,000 people wake up, wherever they are whichever time zone, Louisa has visited every single water cooler at the company, and gives you all the gossip that you'd want to know by bumping into a friend or a colleague, which is so and so left the firm so and so got promoted, so and so's now covering IBM, someone's just moved from Hong Kong to Singapore, whatever the thing is, that's automatically surfaced to each individual on the people they know, within the firm. And we know that from the metadata based on who their zoom calling with. So suddenly, you get a little summary of your people, so you don't miss a beat on your people. And that just helps spur essentially collective intelligence, which is people, people helping people, people needing people. So yeah, I think I think that's, that's, I mean, a nice blog is that, you know, you've been helping me, you know, we met the other day, and he's been helping me already with, you know, getting AI talent, because he's plugged in, in the university space. So you know, just just how things happen. 

27:19 Maryam 

Yes, I mean, right now, we call it networking and the serendipitous connections and insights that come out of that. It's interesting, the people aspect of collective intelligence to me really brings up you know, how do we involve the different types of stakeholders? And how do we really democratize the decision-making process? I'd love to hear from everyone on the panel of how you're thinking through that within your organization now. 

27:43 Carlos 

Yes, I can start. So I was last year in a panel like that, inside the MIT CSAIL MIT. And a good friend of mine, the guy that really built theory for Apple, he came to, you know, like the stage, and he made the stand statement, everybody was absolutely silent. He was saying, look in the NLP area, we ultimately we've completely failed. for three reasons. Number one, the training process, it's about $10 million, every shot. So it's, it's just a few companies can do that. Number two, we are still committing a lot of mistakes. And number three, we are very far from replicating the human brain on that domain. So that's all about the collective intelligence. Another example, closely related to that I'm working on, it's like understanding inflation. So all investment banks all around the world. They're trying to understand inflation, because we were lucky in a world with no inflation, very low interest rates. And now we completely change gears, only one training process. On one of these experiments that I used to do, it's over $50,000. So it's definitely not democratized. We have a long way to go there. And we need to think about solutions where everybody is involved. The technology is already there. So we can, like have a massive involvement from our folks on that team. We have blockchain, we have distributed systems, we have people that want to collaborate, we have everything, but we still need to go something where artificial intelligence cannot go, which is like convincing people that we need to work in, in teams in groups in a massive ecosystem to get there. That's the missing piece. 

29:56 Maryam 

That's interesting. This is a complete aside, but I heard Hod Lipson speak once and he was talking about how anything negative you say to Siri becomes training data for future series. So be careful what you say to your devices. And I'd love to hear from anyone else talking about the democratization piece. Yeah. 

30:16 Abhishek 

If I may chime in here. So essentially, I think from my I would like to take a jab at it from a golden circle perspective, the wise the Watson house, right? And for a pragmatic approach that's been super helpful. Like why do you really want to create a collective intelligence system? I think the answer is pretty simple answers. Why do we have Google? And why do I have Netflix? Right, you're leveraging somebody else's recommendation somebody else's behavior to benefit you, whatever the task be. So from that perspective, I go into the watch of it is really, the key aspects of it is making it playful and making it attractive, right. So that user face that Google user face user interface of Netflix is seamless. And how many of our systems which are collecting intelligence, how many people can really interact with these models? Right, it's predominantly the developer community. And there is a specific skill set beat by Finn, or by beat SQL or a bunch of other programming languages that is enabling these people to interact with these collective systems. So human machine interfacing is really a key piece of the puzzle here in the in the kinematics of this process. If you make it a super easy experience for me, I'll go and do it. But with that, I want to mitigate my risk, I don't want to go get a very risky answer and then execute it to realize it's, it's not something I really want to do it. So I think it's a balance of really creating an easy user experience while mitigating risk. And I think it'd be in why we call augmented intelligence before artificial intelligence, because augmented means you combine the human machine, and all the XYZ participants is weight age to determine what your final outcome is. So I really believe it's going to be a combination of how simple the tools are, with the combination of how risk friendly your outcomes are, because, I think the easiest example I can tell you is the regulation one, if there's a new regulation in EU that needs to be applied to 20 different products, obviously, everybody wants an easy way to interact with it. Otherwise, you're really creating a choke point between the expertise group that knows it, right. But you don't want a scenario where a decision has been put out, or a result on Google is coming out that's dangerous to the process you're doing. So it's really an art today, I would say. And I would really say the power is coming from designers more than the developer community here, designing the right kind of projects, designing the right kind of interactive systems that enable a sustainable growth. Because I mean, if you've put out ML projects out there, you get a lot of momentum, you got a lot of acceleration in the early stage of it. But how long is it sustaining. So that's where the designers are become the key players. And the power has to really shift from some of the development community, to the product community and the consumer community, because that will really transform the systems into more glamorous systems. I'm sure Netflix is not the most glamorous systems when you once you go into the micro services they have, right. So I really believe that the persona transformation that can change the way we approach collective learning and how it really gets democratized. It has to be sustainable. 

33:39 Maryam 

Yes, absolutely. Balancing the UX side with the trust element for the new type of user is going to be important.  

33:46 Cynthia 

Yeah, kind of following up on what you were talking about. You said that the designer really plays a big role to make this stuff accessible. And I think that the developers and researchers need to step up a bit too. So for example, one of the projects I'm working on is there are so many anomaly detection methods out there, tons and tons and every year, you get more like, how do you make your choice, if you are someone who doesn't do research in this space, and you're not a scientist, but you want to use some of this for your data, you're overwhelmed, you're intimidated, like you don't even want to play with this stuff. So one of the projects I am assigned is to find out based of the data that you give me, how do I choose the right anomaly detection method for you? So I basically take that worry away from you. Now, people are more inclined to use the stuff that's what we should be working on. We should be working on making this AI as cool as stuff that you find in research papers, and conferences more accessible to people because otherwise you do all this research. It's not being used. What's the point? Right? I mean, cool. You get an on your resume. But if you're a research scientist, but if no one's using it So just kind of following up on what you were saying. 

35:04 Maryam 

So yeah, having a system that takes into account the constraints before you do all the research and run into the dead end would be key. Kriti?  

35:12 Kriti 

I think adding to some point. Yeah, that that's very much how, you know, my team likes to think about what we're doing, as well as that. It's not even just the AI ML components, but even simple things like just having insights, right? Like just taking regular statistical insights, how many people are going to go and download their data and then like, put it into Excel, and then try and track like, where should I be spending more money in terms of my marketing spend? That's not really, you know, they're, they're busy doing other things. They're busy building their own business, they're entrepreneurs. So what we can do is just by democratizing, not just the AI ML component, but the data availability, and the insights that you can generate from that data, just that by itself is a huge plus. So you can make better decisions based off of, hey, just statistical simple, like averages, moving trends, you know, where, where am I going to optimize for my next decision, and that's something that we can provide, in addition to, like you said, like, the algorithms that again, you know, like, if I had compute resources, I could build out the best Chatbot. But probably, as a small entrepreneur, I don't have those kinds of resources. I could build out marketing emails, and spend my entire life wordsmithing, like this word will, like, get me an optimized response from a buyer. And that's kind of one of the things that my team really tries to focus on is like taking that and abstracting that away from, from merchants so that they don't have to think about these things that are not their core job, they can worry about building their business, they can worry about other things will take care of like generating the text for the perfect marketing email, we can worry about generating your chatbot for the perfect response. Somebody's asking what's the shipping time we'll provide that without a human having to step in and spend time on that, when they can be spending time on higher level things, like building out their business? 

37:33 Maryam 

Interesting. And I feel like people nowadays can use things like ChatGPT for creating the perfect email. But I love your idea of you know, obviously, the end goal is to have a user-friendly system that non developers can interact with. But even what we can do now is just being more transparent with some of the insights we're providing. That's great.  

37:55 Theo 

Yeah, I think in healthcare, the democratization is key, but it's also something that needs to be done in a very careful manner. I mean, the risks in a model that doesn't work well, is not that you're going to buy the wrong thing, or you're gonna watch a movie that you actually don't like, but you might harm a patient. So having tools that are available to physicians is, is key, of course, for them to use them and and making it available to clinicians that might not have access to a team like ours within Northwell Health, right. So a small hospital somewhere in in rural US or somewhere else in the world. The issue there is that these models, first of all, are not great in their transportability, because they're built on local, mainly data. They will work well for New York, but they might not work well for Africa or China or Europe. And there's always that need for either retraining or recalibration or making sure that you're addressing, let's say the local differences, both in terms of populations, but also in the way that healthcare is delivered. But it is important to provide these tools and more importantly to really educate their clinicians of both the capabilities and also the things that they should watch out when they're trying to use such tools. So I guess collective intelligence there is key and it might be better than you know the usual sources of information that they're getting. We saw this this example early on Covid where our frontline workers were scrambling to find any way of treating these patients. And they were going to Twitter and they were looking for tweets from their German counterparts. And then they were taking information from like, a random Doctor somewhere in Germany that were saying that, Oh, you should look for this specific lab, because that's going to be key, like that's over 10, your patient is going to die. And, you know, we started getting these information from our clinicians on, Yeah, we should be looking at this. So we ran the numbers in our population, it just wasn't true. So collective intelligence, when done right, can address all these different issues. But again, the framework needs to be there to make sure that you know, where when we're democratizing our models with, do all the checks on the fact that they are well validated, they're transportable, and they also maintain their performance throughout drifts that can happen during time. So 

41:05 Maryam 

yeah, having a good idea about the parameters before you decide to retrain it on your own population data. Roman, 

41:12 Rohan 

yeah, I was just gonna say, you know, building on what Abhishek was saying about, you know, net Netflix beings, you know, seamless. And, Cynthia, you were talking about, you know, without people using it, what's the point, and more importantly, without people using it, you don't get enough data. So it's like a death spiral, as opposed to a virtuous one. And, actually, you know, you were saying how, you know, so people can spend their time doing other things, or you watch Netflix, but anyway, the point is, when we built this collective intelligence platform at Goldman, where you could say, hey, who at the firm can help with inflation swaps in France? So, farm financing in Australia, whatever it is, we built it. And then we, you know, build it, and they will come? They say, right, so we built it. And, you know, they came, I mean, we started with 500, searches a day went up to 1200 searches a day, but it wasn't quite what we thought, we thought it would be way more. And then we came up with some other ideas, we said, look, we've got a map of what and who everyone at the firm knows. So why don't we read a million articles a week from 250 global newspapers, which will take any human long time. So every morning, we read every week, we read that, and then we send information to the right people at the right time. And all of a sudden, our usage just started like just rocketing, because now all of a sudden, every person comes into work. They get an email from Louisa saying, Hey, you cover BlackRock, Apollo, Blackstone, so you get headlines on what's happening with them in the world. And if you click on it, you see, oh, Blackstone's entering semiconductors in Taiwan. So you know about Blackstone, but you don't know anything about Taiwan or semiconductors. But there's six people at your firm that do. So all of a sudden, it became a cross selling machine. And, you know, it's every CEO, every business, collective intelligence is cross selling, coming up with each other's ideas, and, you know, selling it and upselling and cross selling. And so all of a sudden, we had, you know, 12,500 people a day like 12x, you know, coming and and we've closed several multimillion dollar trades as a result of just connecting the right person at the right time, because of this systematic cross selling machine. And so, so that's just an example of like, you sometimes You've almost got it down. You just need a few more iterations of the Rubik's cube or whatever. And then and then it's there. And so, so yeah, that was one of our experiences. 

43:57 

Awesome. Thank you, everyone. I think the just in time knowledge is makes a lot of sense to really harness the collective intelligence and make it actionable as we were talking about before. I think that brings us to the end of the panel. Sorry, we didn't have time for questions. If you do have questions, come follow up with us and ask it after the fact. But thank you so much to our panelists today, Theo, Abishek, Cynthia, Kristi, Rohan and Carlos, thank you all today.