Introduction - Paola Santiago
🫛 This edition of AI, Software, & Wetware features an audio interview on ethical AI with Paola Santiago, a 🇨🇦 Canada-based Fractional Product Manager for startups and scale-ups. Paola is the author of the PS Product Person Founder Series newsletter on Substack. We discuss:
Running co-working events at local coffee shops to help startups and founders find each other and solve implementation issues
How her early experiences with the Stardew Valley game inform her work as a fractional product manager
Why she sees a resurgence of marketing and community perspectives overtaking the impulses to put AI in ‘everything everywhere, all at once’
Developing a prototype enterprise SaaS tool for a client within a few weeks to get into beta test quickly for customer feedback
Concerns about Epic Systems having detailed personal data for 300 million US-based patients, vs. siloed Canadian provincial health systems
and more. Check it out, and let us know what you think!
This post is part of our AI6P interview series on “AI, Software, and Wetware”. Our guests share their experiences with using AI, and how they feel about AI using their data and works.
This interview is available as an audio recording (embedded here in the post, and later in our AI6P external podcasts). This post includes the full, human-edited transcript. (If it doesn’t fit in your email client, click HERE to read the whole post online.)
Note: In this article series, “AI” means artificial intelligence and spans classical statistical methods, data analytics, machine learning, generative AI, and other non-generative AI. See this Glossary and “AI Fundamentals #01: What is Artificial Intelligence?” for reference.
Interview - Paola Santiago
Karen: I’m delighted to welcome Paola Santiago from Canada as my guest for “AI, Software, and Wetware”. Paola, thank you so much for joining me on this interview! Please tell us about yourself, who you are, and what you do.
Paola: Thanks Karen. I am so excited to be in this interview with you. My name is Paola. I am a fractional product manager. I’ve built a consulting business, basically a product studio that helps startups and founders and organizations, from growth marketing to product management strategies and services, including roadmapping and making sure that you’re building the right thing, to even technical implementation.
So you built the right fit, you built the right product, but only for beta. And we want to make sure that you are safe in terms of cybersecurity, and your database is scalable, and you can support more people or more users for your product.
Karen: That’s great. Yeah. When I was working in corporate and software development, we had a phrase talking about verification and validation. Part of it is, “Did you build the right thing?” And part of it is, “Did you build the thing right?” And that’s where qualities like scalability and security come into play. So it’s great that you’re focused on that.
Paola: Thank you. It’s the technical implementation side of it all. You know, I built software before AI was even accessible for most people. And the software development lifecycle has a process for a reason. But now with AI and the ability to build an app immediately, that type of checks and balances are kind of getting buried. And that is introducing quite a little bit of a problem into the ecosystem here.
Karen: Yeah, so I hope we’ll be able to talk more about what you do and how you do it. One thing I’ve found really interesting is you look at all the people chatting about how they vibe-coded an app and didn’t get what they wanted. And part of it is prompting better. Basically, prompting better is equivalent to writing better requirements. If you think in the terms of the traditional software development life cycle, that’s what it is.
Paola: Absolutely. Oh my gosh. We can talk a lot about that. I’m doing a workshop. I’ve even done a small mini-hackathon for people. But the whole prompting is based off the actual analog life of creating a PRD, a product requirements document, which usually includes a design aspect and technical feasibility reviews in here. So I agree on that.
Karen: Yeah, great, I’m hoping that we’ll hear more about that. You talked about running some events. Can you say a little bit about what you do there?
Paola: Yeah. So aside from running this studio to help startups grow their user base or getting revenue, I also run a couple of co-working events where we invite startup founders and builders in this sort of very relaxed and chill space. Usually we ask a coffee shop from 5:00 PM to 8:00 PM, “Hey, chances are you won’t have a lot of customers. Is it okay if we hold an event there where people can come in and work on their projects?” So people who want to work on their project after work can come and meet people who are in the same boat, what are they building? If some people are trying to look for co-founders, maybe they’ll find it there.
And we can talk about this, but I find that it’s not just me who’s doing this. It’s actually a cultural movement. There’s about three to five coworking events happening. So even in the life where most people want to be remote and working remotely, they want a community. They want to meet people who are kind of doing the same thing. And even in the life of AI, everybody’s using AI in these sorts of events, yet they want people. They want to be near people who are doing the same thing and want to get feedback from true builders around them about what they’ve built, and any resources that they might have. Like, “Hey, have you looked into this type of MCP? Or “Have you looked into using Cursor instead of Lovable or Claude Code? What’s the differences?” That’s the coworking events that I’m holding.
And also, I interview startup founders, kind of understand the human side of building a business and building a startup. So I’ve been working with highlighting startups in their, as much as possible, early days. Or at least for people who can remember the early days, so they can share about their experience building a product from when they have limited funding or when they have to think about finding the right partners or co-founder or team member. I remember talking to one founder saying “It’s always going to be a people problem.” So I think that’s huge. I keep hearing these notions to replace people from AI, but at the end of the day, it’s still a people problem, and people, you can’t just replace them. So I’m running these events here and it really highlights how people are just not replaceable yet. Or, at least, definitely not right now.
Karen: Yeah, I hear a lot of people talk about these ideas that you can completely replace whole teams of people with an AI tool and that’s just really not effective in so many cases. Having an AI that augments your team and helps them to take care of the drudgery is a much better solution. But I’ve heard a lot of people talk about that.
I do know a founder who might be interested in talking with you. She’s working on something called Space Situational Awareness. Basically, satellites not running into each other in space.
Paola: That’s so cool.
Karen: It’s very cool. And she’s a neat person. I think you’d enjoy talking with her. So remind me to introduce you to Amanda when we’re done with this call.
Paola: Going to put that in this note right here. Yeah, no, I, I love that.
Karen: I should make a note for myself. Introduce Amanda. She’s still very early days, so I’m sure she’d love that.
But yeah, those sound like neat events. There are actually a lot of startup founders, entrepreneurs, solopreneurs on Substack, so you hopefully will find some people there from this interview that would be interested in sharing their stories.
Paola: I’d love that. Yeah.
Karen: Yeah. Well, great. You mentioned writing software long before AI came along. Can you talk a little bit about your level of experience with AI and machine learning and analytics? It sounds like you are probably using it professionally, maybe personally as well.
Paola: I mean, before AI it was really machine learning and data analytics. More than 10 years ago, those were the buzz words of the time. So during my master’s, I was already implementing a lot of these tools so that I can understand the fundamentals of it.
From a sociological background, for example, I’m very interested in sentiment analysis and extracting words and understanding how people speak with one another, not just through what they’re saying, but how they’re saying it, and the emotions that come up into these words and into these paragraphs. And that was my thesis during masters and I was able to understand.
A funny thing is: I was really addicted to playing the Stardew Valley game a long time ago, and I was reading a lot of Reddit, for example, about the game. And that helped me understand why people love this game so much. Like, “What are people talking about?” And that type of sentiment analysis is what most companies, especially nowadays, are trying to replicate in terms of community, in terms of being able to empathize with one user. And so that’s the fundamentals. How to do that is what I tried to learn early on, and then during my actual professional career.
So for example, I was helping build this property management platform in the states when I was living there. The biggest problem was that onboarding was taking six months to train the employees to use their management software. And aside from, of course, lessening all the friction points and creating a great user experience, it’s about making the search bar as easy as possible, versus using search bar right now, or even back, especially back then, Microsoft Windows – it’s still a string, for example. So it doesn’t understand your intention. But it understands the words and it matches the words.
So for example, I say “Google Chrome”, then they’ll know to go to Google Chrome, because there’s a Google Chrome there, and the word Google Chrome is there. But, say, I want to search this. I want to sign up to make sure that I get this discount for my birthday. How does the search bar know that? And that is where the whole AI sentiment analysis and training start to come in – training the AI to understand what is the person’s intention, what is the employee, what are the user’s intention, and what are the types of features that would go along with that?
And that was technically before OpenAI really burst to the public. There were already professionally-made tools for it. But it definitely wasn’t as sophisticated as it is now. And the explosion of it, I think people are still trying to wrap their heads around it. I will say that the big companies, if they are security-compliant and they care about liability and legal issues, most of them still would not put AI, at least on the client-facing side. They might start to implement AI from an internal team’s perspective, but not a front-end perspective. I’m sure that there might be some, but for example, banks using AI would be to show a person “This is how much you have in your bank account. These are ways to improve your credit score based off your analytics.“ Or just tell me what to do and I can give that report to you.
That is a very dangerous situation or dangerous play for many companies still nowadays because of just the hallucination. AI is still not sophisticated enough and from the computer science perspective, there’s many engineers who believe that the way LLMs are being talked about right now won’t actually get us where the promises are that we hear about in the future. That’s a huge sentiment right now as well. It’s not in the mainstream sentiment, but it’s definitely within the developers. There’s the developers who are building it for the products, then they’re fine with it. But the big companies that have trillions of data, they’re the ones with hundreds of millions of users. It will be very, very difficult for them to properly implement AI without the liability issues at the moment.
Karen: Liability is a big concern. And it’s interesting that you work a lot with startups. I was just saying to someone I talked to earlier today that one of my earlier guests, Julie Rennecker, had commented that founders tend to imprint their startups with their values, whether they realize it or not. So if they value being in compliance, and ethics, and respecting people’s privacy, then those values are there from the beginning. But if they’re not, it can be very hard to retrofit them onto the company and the product afterwards.
But yeah, in the bigger picture, there are a lot of people – you mentioned it not being mainstream, but Gary Marcus has been saying for many years that LLMs were not going to be the path to get us to AGI, that it just was inherently flawed. He was advocating for something called
neurosymbolic AIas a better path to help to get us there. He’s been shouting into the wind and really not being heard. And then suddenly, recently, some other prominent people in the field have started adopting. “Oh, you know, guess what? This isn’t going to work.” He’s been telling us that for years and people are just now finally starting to listen.But yeah, the whole liability perspective and putting it between you and your customer and not really being able to predict what it’s going to do or tracing it – there’s a chapter actually in Volume 1 of our book, by Elena Calvillo. I don’t know if you’ve read her articles, but she writes about ways to try to at least control, almost like version control on the prompts and having more structure around the way that you integrate AI features and making sure that they’re repeatable. They change the model, and things change, and there’s so much variability there that we don’t control, and that could really burn a company.
Paola: Oh yeah, absolutely. I want to read that article actually. It would burn a company. Maybe not the smaller ones at the moment, not the ones who are popping up at the moment, but eventually, once they start making revenue. Yeah.
Karen: You mentioned banks, the financial decisions and recommending people what to do to improve their credit scores. They could give bad advice. Another concern there is the biases that are built into the way that it analyzes and makes recommendations to people, and who they recommend to give a loan to, and all kinds of other dimensions of it. So a company that put that kind of interface between them is kind of taking those bias and hallucination risks on themselves. And, you know, just because an AI told you to do it doesn’t mean you as the company aren’t responsible. That’s probably something you end up talking to people about quite a lot?
Paola: Yeah. Actually, now that you mentioned that, there is a pretty big financial institution and, if I remember, they were talking about the Apple approach. Because we don’t notice, but Apple actually uses AI, and we just don’t know that. They use AI into a client-facing side or the customer side, by using language and manipulating language in a way that gets them off the hook in terms of liability. So instead, it would be like a suggestion, instead of “This is your report. This is what needs to be done. This is based off your data. Maybe you should do this. Based off data, this is the best next step.” It would be recommendations. “Have you looked into these types of options?”
And those are very distinct ways to go about it. because they still want to train. I think a lot of companies are pushing forward with this LLM model no matter what. It doesn’t matter until they hit a huge block, or until another alternative comes with not an LLM as you mentioned, this other one for the AGI?
Karen: The neurosymbolic AI, as a path to AGI, is what Gary Marcus talks about. Yeah. That’s still in research though. It’s coming.
Paola: Until that research becomes viable, I think people will continue to move on to this LLM, it doesn’t matter what, and the risk just becomes a liability, because the money is just way too lucrative for a lot of them.
And the competition just became a lot more strict. Because now, unfortunately, especially for people working in general, it’s, “If you don’t use this, then ...” For example, I’ve heard some software engineers – very, very talented, but they refuse to use AI, and they were fired from their company. And that’s the sort of situation that we’re in. And I don’t know whether it’s correct. I don’t think it is. But there is a craze at this moment. It’s still pretty chaotic for everybody.
Karen: Yeah. There are stories about people being told that, “You must use AI”. And companies, I think Meta monitors how many tokens people use per month, and if they aren’t using enough. Okay, you want them to run up a bigger bill? But there should obviously be a balance there. And it sounds like you are trying to help startups make better decisions about how and when they use it, which sounds very wise to me.
Paola: Thank you. I mean, people just put AI and stamp on it. I was like, “but you don’t need AI for this.” I don’t understand, actually; it makes no sense. It is a craze. I think the stat from this year was that almost 70% of new tech companies, if not more, are currently AI-driven. And from a startup perspective, 90% fail already before AI. But now there’s also another stat, and I’m sorry I can’t for the life of me remember, but it’s a pretty high percentage of how most of these AI companies don’t even make revenue.
And that’s the other part of what I am trying to move forward with. This studio that I’m working is: I’m telling people AI is a tool. Back in 2015, competition was fierce, because you have the right talent and the right talent will get you that feature built quicker, versus another company who don’t have the right talent, who have a hard time replicating the features that’s needed to solve a problem. And from that perspective, users are just going to use your product because you have a better feature, a better company. Whereas now that gatekeeping or that moat isn’t necessarily there anymore. The talent isn’t really what people are looking at. It’s now people, again. It’s more about the marketing again.
And I think there’s a resurgence of marketing and community perspective – people who want to use your company because they believe in you more than more than the other one. Now, the same companies are probably providing the same things, enough for people who don’t really care enough. So another one is onboard. So those who’ve been in the game for a long time, they have the advantage. Their biggest advantage is your onboarding problem.
So I’ve kind of switched a little bit back to growth and people as well, because I see that, while everybody’s focused on the AI and the new tools. And I find tools like those, they’re for the most part very similar, and just a little bit different, just a little bit better. But at the end of the day, it’s still the people that’re going to get you where you need to go. And that’s what I’m trying to put out there for people and be like, “It’s not just the AI, it’s how you use the AI.” It’s also how you build the actual people around you to support your product versus others. And that’s kind of the dichotomy. And also, just because you built it doesn’t mean it’s scalable too. So that’s the other technical aspect.
Karen: I’d like to hear a specific story about how you have used a tool that has AI or machine learning features, and maybe share some thoughts about what went well? And then maybe an example of something that didn’t go so well?
Paola: Absolutely. It’s funny. So when I started this business last year in 2025, I dove right into the deep of AI, like deep, deep into it. Before Claude Cowork was even a thing, or Claude Code was even a thing, you were able to kind of create a hack with that, with Cursor and through an MCP and all, and try to create products. I created basically one of my big accomplishments. I used AI vibe coding from Lovable to create an enterprise SaaS tool for a client by myself. From my perspective, this would’ve normally taken me six months of work with a team of five people. And it should have cost them 300K minimum just to get a beta, and they would only get it in six months. But with Lovable, and with a friend of mine who helped me figure out how to transfer the lovable code into GitHub and then cursor with Claude, I was able to create a very sophisticated tool that connects both employee portal with customer portal and basically a good enterprise SaaS within I guess two weeks, I would say. And that allowed us to get beta testing in no time, honestly. Maybe two months because there were a lot of bugs.
And that’s when I also learned and realized the limitation of this sort of model, especially for traditional founders or non-technical founders. I am privileged enough to have a lot of developers as friends, so that I could get some consulting done for myself and understand, why is something not working? But even with that, you come to a point where I felt I cannot support this anymore by myself because this is going to require an actual team of software developers, especially now that we’re trying to come to dive into the revenue aspect. I started as a product manager in this role, and I ended up being a technologist, like, head engineer without really an engineering team, like a software engineer in this project without the proper knowledge of coding. And I understand my limitations around this.
I remember telling clients around this, so I’m just like, “I can only get you here. My role was supposed to help you get to beta testing and understand customers and what they think and what they’re doing.” But what I ended up doing is understanding the process as a business analyst perspective. Look at the process, create a SaaS from it, maintain that SaaS platform, connected to Stripe, connected to a bunch of other third-party applications. And understand, in order to scale this, you need someone who understands how to scale this with a database. I was able to create a database. I tried to make it as secure as possible. But at the same time, I don’t have the ability to truly do penetration testing, for example. I can only QA properly.
And so I think what I’m trying to say is that it was a great experience understanding how, if I had this when I was a product manager, we would’ve finished things a lot quicker. I would’ve been able to provide a lot of A/B tests and not bother my actual software engineers and let them deal with it. And I could have protected my software developers from the marketing or sales, because I’d be like, “Look, I’ll build you the tool.” It’s fine. I could do that, because chances are you just need something quick and easy. And of course now the agents are coming in so you can automate a lot of stuff as well. So you can build those things. N8N was the big tool last year, and now it’s being replaced by OpenClaw, basically, for example.
So what I was trying to say is I did a lot more tools. I worked with a lot more AI tools last year than I do this year, mainly because from my experience, I’ve kind of seen my limitations with it and the limitations with the AI that I have.
I also tried to use AI for marketing, for example. That’s a big one. Creating content, creating all these things, help me create a script and I would video. And I did 30 days on TikTok as to what I would talk about with AI. But there were so many things that were wrong, if I research about it, if I truly look into it, where I used B-reels that were AI-generated. And that was something from someone who’s never really done video editing before. I really didn’t really want to put my face out there. It was a great alternative.
But then, the whole Studio Ghibli thing happened. And I realized that it started to become eerie, that my brand itself became eerie. It became like, “No, I don’t like this path that I’m going to, when it comes to using AI from that perspective.” I can’t. Although I can’t fault those who do, especially small businesses. It is expensive. There’s a huge sort of pressure for many businesses to stay active in social media, to be aware, for people to be aware of them. And there’s just so much noise out there.
So I understand why they would use it. I personally found it quite – uncanny, I think, was the word. And so I stopped using it. Because I used so much of AI last year, I learned where it becomes uncanny. And I don’t like it, to the point that this year I use it a lot less.
So, for example, when I create posts on LinkedIn, I really do take 30 minutes of my time just to write a post, which is so annoying. But that is a huge time for me. And I will still use AI in terms of review and editing and grammar. But it’s not to generate what I actually like, my ideas itself – my Substack, for example, like writing long articles. I really emphasize on making it as human and with my personality as much as possible, mainly because last year I was able to go on the extreme side and understand personality. Now I see all the LinkedIn posts, even in TikTok or Instagram when they speak, I’m like, “That’s AI-generated script”. You can absolutely tell the personality isn’t there. You can be neutral, but it’s not even that neutral. It’s a very particular type of speaking, of a script.
Karen: Some people call it hollow, like a shell.
Paola: Yes. I think that’s a great description of it. Very hollow. Hollow is a great word.
Karen: Yeah. It sounds like you’ve learned some good lessons about how it works for you and when it works well and when it doesn’t. That sense of the uncanny valley. The tools have gotten better, but there’s still that sense of hollowness. It’s maybe artificially smooth, even, in some cases. I think people had to get a sense. I don’t know if you saw – LinkedIn has had this thing for a while where they “Let us help you write with AI”. And I think, “No, no, no, I’m not going to do that.” But Substack just started – I saw a post. I haven’t seen it in my own Substack app yet, but apparently some people have this, maybe a trial feature of having Substack propose generating some AI-generated comments to reply to a note. @Sam Illingworth wrote about this. I’m like, “Ew, I hope that beta test fails because I don’t want to see more of this.” I mean, the whole point of Notes is to actually engage with other humans. My response was literally, “ew”.
Paola: Is it like they create an AI-generated comment?
Karen: Yeah. I’ll find the link for it. A few people have it. One person tried it on Jennifer Hong’s post, and he shared what it gave him in a thread on her post. And it’s like, “Ew, I don’t want to read any of those.” So I hope people don’t do this.
Paola: It’s so sad because I see why people would use it, but at the same time, it is so sad that we’re going into that direction. I see a lot of people use that. I can see a lot of people using that for LinkedIn, for example, especially those who are job hunting now. It’s so difficult to get a job, for most people. And they need to be known in some way, one way or another. And one way to do that is commenting. But most people are lazy and it takes them a lot of time to do it. So I see the other side, but also from one end, it’s just like, “Oh my God, there’s more bots.” I’m trying to figure out, like, I can already see bots on Instagram sometimes too, and it ruins a lot of the experience for most people. And Reddit, most of the people in Reddit are not speaking the same way as it was 10 years or 20 years ago. And it’s just, it’s a sad reality.
Karen: Yeah. Yeah. It’s funny because, you know, these AI tools were trained on us, but they’ve kind of drifted toward the average and the personality seems to get lost. But then the other thing that’s happening is, on the flip side, as people read more AI-generated content, we’re, I think – maybe unconsciously – drifting towards picking up some of those habits and those verbal patterns. And we’re going to start sounding more like AI, not just AI sounding more like us.
Paola: I know. So sad. That’s not great. I wouldn’t prefer that.
Karen: Yeah. You and I both don’t use it for our writing because we want our personal views and voices and styles and everything to come through and sound human. Some people, as you said, use it for spelling and grammar, and I don’t fault anybody who uses it for those purposes, especially if they’re not writing in their original language that they grew up speaking. I mean, how can you criticize somebody for that? It’s almost ableist, you know, to do that. The point is for the person’s ideas to come through, and if the tools help them do that, then it’s very hard, I think, to criticize that. But at the same time, you know, pushing a button, something pops out, you post it – that’s not adding any value for anybody. It doesn’t make the person look good, and it doesn’t add value to someone trying to read it. So I don’t get that.
Yeah. On the flip side of that, LinkedIn had this, I guess it was summer 2024, where they came out and said they were going to start using everything we had put into LinkedIn up at that point. They were going to use it and they would keep using it unless we went in and actively opted out. Did you hear about that when that happened?
Paola: No, I didn’t hear about that.
Karen: Oh, I’ve got to get you the links. There’s two places you have to go if you want to opt out. And I’m not saying that you will or do, but people should know that this is something that they’re doing. And same thing with Meta. I’ve never used Instagram, so I don’t know exactly, but Facebook did kind of the same thing about two summers ago where they said, “We’re going to start using your stuff.” Yeah, you know what, if you’re not going to let me opt out, I’m going to go delete it all.
But there’s a lot of discussion around where these AI tools get the data and the works that they train on. They use information we’ve put into online systems like LinkedIn or Facebook or that we’ve published online places like Substack or other articles. And I’m wondering how you feel about companies that use this information for training their AI tools to something they scraped from the public internet or from other sources. And do you feel like people should have the right to consent or not to whether or not their data gets used and maybe even to be credited and compensated if it gets used?
Paola: Ideally, yes, ideally that would be a great ideal world for us to have. The EU, I think, is the one who really started this first with GDPR. Scraping of data from the internet, that’s been done before AI. It’s not even for AI, it’s just been done in general. Because there is a lot of valuable information out there, business-wise, that’s needed. And from an ethical point of view, yes, people’s data needs to be protected. People just need to be protected. I just got a phishing scam earlier, for example.
Karen: Oh, wow.
Paola: I’m pretty sure that was a phishing scam. I don’t know where that data leaked or who leaked that. And I can fairly say maybe I’m a little bit more tech-savvy than the average person. And so for me to get these types of scams is already telling. And I can most likely protect myself versus those who can’t. So in an average sense, yes, I do wish that there are ways to compensate for this.
Just also being a devil’s advocate to myself: like I mentioned, I also understand why companies need this, because they would not be able to grow without them as well. I’m not talking about the big companies, actually. I’m thinking about the smaller companies who bootstrap themselves, who don’t have money in the beginning. They might not even pay themselves a salary for a year or two. And I’m thinking about those types of companies. And the only way that they can really survive is through free use of data or whatever is free. Free, it’s a bit up to interpretation on that end and understanding how to scrape data, depending on their industry. Because they need someone for cold email. They need someone just for a sales perspective. They need to understand who to email to.
So it’s really difficult because I want the consumers to be protected. They deserve to protect artists. I think where I’m really drawing the harder line is at artists, especially with their creative works, and saying that I also just understand from small businesses point of view, like a lot of startups, they have no capital to work with.
And there’s already huge companies out there who, you can’t compete with them in a way, because they’ve been there for a long time. Unless you do these types of things, I don’t even think you can build a viable company until you have enough data. I know very amazing startups. They’re ethical. They really don’t want to use their user data, or they don’t want to look into the personal data, which is honestly the most valuable part of it. But at the same time, they’re struggling with their runway because they have a hard time. They don’t have enough people or resources to continue moving forward, and they don’t have the deeper information, the deeper analysis that they need, based off the data, to choose the right direction.
So there is that type of balance. And I personally have a hard time understanding what is the balance, and maybe I’m hypocritical in one way and not in the other. I would love to say that every person, every consumer should be compensated, especially when they’re giving data away. I wish that maybe there’s a balance where if a company’s making this much revenue, then they should be the one compensating. The ones who have no money to start a company, they would never be able to survive.
And then you don’t have the competition to break up a monopoly. So I’m trying to be balanced. I think at the end of the day I’m balanced about this, but it’s hard. I want to say yes. I want to protect everybody. But also I don’t think that’s also very practical unless you think about ways to really say, “Hey, look, you’re making $500 million. Your profit is 150%. Maybe you can bring it back.” But even for that, that is too socialist for some, and that would discourage a lot of innovation, apparently. So it’s also that type of problem. The ones who are innovative are the ones we have without money anyway. So, actually, we’re giving them a leg up. That’s a conversation for another day, I think.
Karen: Yeah. So the companies that are getting this data – now let’s maybe focus on the big ones – do you feel like they are transparent with you about where they got the data for the tools that you have used?
Paola: Oh, absolutely not. And they know that, absolutely. Has there been one?
Karen: Do you know of any that you feel like have actually been transparent with you? Big or small?
Paola: Like I mentioned, there’s a startup that I was helping and they were very transparent, not using their data, so that was really lovely. But as I mentioned, they’re also struggling. There’s that side of it. Yeah. But another company who’s very transparent? I can’t think of one right now, unfortunately.
I’m sure that the cookies one is a nice touch, but I think that took years. That started with the EU. When I lived in the UK for that time, I was like, “Why is it, every website I go to, there’s always these cookies?” But now every website I go to is not cookies. So I think that took a 10-year thing. So I think that’s the closest in terms of transparency. But that came from a legal perspective. That’s not out of the company’s values, heart, necessarily.
Karen: You mentioned that you had what you thought was a phishing attack. Was that email or phone?
Paola: I think it was an email. It was an email, yeah. Yeah.
Karen: Do you know of any cases, other than that, where your information has been used? Or it showed up somewhere in a search, showing something that you hadn’t given away? Or anything like that?
Paola: It is usually my friends who would tell me, “Hey, didn’t you do a 23 and Me? You’re going to be compensated. They just breached data.”
Karen: So you’ve had data breaches that you’re aware of? Because I get at least one a year, and it’s been a couple this year already.
Paola: Absolutely. There is a website that shows you chances of you getting a data breach. I think, Wendy, when you put your email, and I think I’ve had 41 websites where I put my email and they’ve had a data breach, for example.
Karen: Wow. 41, that’s a lot. Wow. Okay. Yeah. If you have the link for that, we can put it in here and maybe some of our audience might want to try that site for themselves.
Paola: Yeah. I haven’t used it in a while. But, definitely, I’ll find it. Let me take a note.
Karen: Yeah. As far as usage of our personal data, any thoughts about that? Any examples where you feel like your information might not be getting used appropriately?
Paola: Yes. I found out about this because I was working with a client for a competitive analysis part. And I found one of the competitors is this huge healthcare system in the states. I think it’s Epic Systems, and they have three AI tools now, but they’ve basically trained their data from like 300 million patients, you know? I mean, when I lived there, I’m sure my data’s in there. My clinic was part of Amazon, so I’m assuming it’s somewhere there now too.
I was talking earlier about institutions that make no sense to have AI with the front-facing clients, financial institutions, what we talked about. Healthcare is especially one that, first of all, I don’t understand how you can have 300 million patients’ records for one company that could be trained with. A lot of the systems here, for example, the healthcare system in Canada, the province, it’s a double-edged sword, honestly. The provinces have their own healthcare system, so the data’s not even talking one province to the other, which is pretty inefficient for some things. But that data is not going to get out of the hospital. Even clinics have a hard time getting patient data from hospitals. There’s a lot of checks and balances with that approach.
And so it’s inefficient in many ways, because it’s harder to properly assess someone. But at the same time, I understand that the patient is well protected in terms of data. I think in these types of cases, it’s hard to fully say I’m against it, but it’s also hard to fully say I’m for it, because there’s just multiple reasons as to why things should work that way.
But I am concerned when there’s one company that has 300 million patients. That’s a very vulnerable pain point. That’s very sensitive data. If anything, that’s the most sensitive data that someone can have is their health, and especially in the States, where that can be used against you in a lot of ways. Just from an insurance perspective, any breach of that. Not even a breach, but any ways that could be brokered and sold to certain companies without anyone knowing. That is probably the scariest way. I understand that there needs to be a way to make this efficient. I like to think that regulation is a good thing. But I understand a lot of people hate that because it provides more steps than anything, and it’s not direct to profit.
I’m going to go step back with the sociological perspective here. And in the sociological perspective, there’s always three big ones. One is the very laissez faire free market. The market will sort itself out. Like, it doesn’t matter if it’s a breach or whatever. Eventually, that breach means that people will be against this, and they would be reprimanded in one way or another. Is that actually what happens? Not sure. I can’t tell. I can’t see it.
On the other extreme side, it would be what you would consider a socialist aspect where every little thing has to be regulated. So the government steps into every single way. They provide money, but the innovative people will have to submit to whatever they say. And there’s times then that might be risky, especially with their lives, depending on who the regime is or what the regime does. So that would be a way to hold these people accountable. Is that how they make these companies accountable?
Versus in the middle – there has to be a balance, but nobody really knows the balance. Maybe most people would say “Yes, there has to be some sort of balance”, but is it 30/70? Is it 60/40? Is it 50/50? And that’s when people get angry at one another and they’re just like, “No, you’re dumb. That’s not going to work.”
For example, should we go back to something where, what we were talking about earlier is after 50 million patients’ records, is there a regulation maybe that you need to have? I don’t know. Yeah.
Karen: The US does have, if you’ve heard of the HIPAA, H-I-P-A-A, regulations which are intended to protect our medical information. I’ve heard conflicting opinions about whether or not it actually protects the family member relationship kind of data, or if it only protects the medical data. But I’m familiar with Epic Systems because they own MyChart. They started out with individual practices. You can get your records from that hospital through MyChart, or this doctor’s practice. And then they got more. And then I think they bought other tools besides MyChart. So they’ve expanded gradually, but it starts out as, “I can get my data from my one doctor here through MyChart”. It started small, one practice at a time, but it’s accumulated. I’m not surprised to hear that they’ve got over 300 million patient records here. Probably everyone I know has had some medical practice that uses MyChart. And there are some other systems which are not MyChart, that actually Epic Systems doesn’t own. So there are other practices out there. But they’re supposed to follow HIPAA. They’re not supposed to use our medical information.
But I don’t know if you’ve noticed this when you were in the US, when you go to a medical practice and they want you to sign something before they treat you, usually it says that they have the right to use your data for “research purposes”. What choice do you have? You don’t get medical treatment if you won’t sign this. There’s consent, and then there’s true informed consent where you have a choice, and I think in a lot of cases in the medical system, that’s not the case.
But what I worry about more, I think, are some of the more invasive FemTech apps with wearables and tracking women’s very detailed health. Those are, I think, even scarier than some of the other medical apps. And the genomics – most of us cannot change our genes!
Paola: No, no. Fortunately not. They try.
Karen: But the idea of giving consent is just so – you know, you can’t understand the terms anyway, so why bother reading them? And most people feel pressured, I think, to just sign. But that’s not really informed consent.
Paola: No. Chances are, as you mentioned, this person needs some sort of doctor, or they need some sort of prescription. They need some sort of healthcare. What else is the alternative?
Is there another way for this? Just, “Hey, I have a business and I need a way for my business to get known from an awareness point of view. I have to sign up to Meta ads or Meta platforms.” Is there any other alternative? Probably. But it won’t give me as much leverage.
And so it’s back to that point where yes, technically there is consent, but how does that get you? So for example, in real life, I just go to Sephora or buy something and they ask for your email or they ask for your phone, and I would have to say, “No, thank you. No.” And then smile.
Karen: Yeah.
Paola: And they’re just like, “No? Can …”, “No.” “No.” I have to say “No, thank you” three times. And they would look at me as if I had three heads!
Karen: Yeah. I ran into that here starting years ago. You’re going through checkout and “Your phone number?” “I don’t give it out.” And they just look at me. I don’t fault the clerks for it. They’re told that they’re supposed to ask. They’re supposed to get that information. But I think a lot of people just feel the social pressure to say, “Oh, they asked me for it. They must be entitled to it. I need to give it to them.” No, you don’t. They need to have a good reason before I give them my phone number or my email address or anything like that. I just say I don’t give it out. And I get some funny looks too.
Paola: Yeah, yeah. Unfortunately I don’t know what’s the alternative of that. Oh, you know what? There is a startup founder, and I think what he is doing is kind of similar to this. The whole point is, instead of companies getting your data, you have the chance to consent whether to give your data, and they will pay you.
Karen: Oh, interesting.
Paola: And that was the whole point of it. I think that’s in theory a very good product to have, a way to combat this in some way.
Karen: Is this a startup that you were working with?
Paola: Yeah, I have to ask him again, because he tried to apply for YC and other incubators, and I wonder if he got into any of them. But yeah, I’ll ask him, because actually he would be a great person for you to interview.
Karen: Oh, sure. I’d love to.
Paola: Yeah. Because there are people who are building this type of counter movement, but it isn’t a lot, and it’s still hard to implement. Because even when I was trying to help him, it’s hard to monetize. When it’s hard to monetize, unless there is a reason, unless it’s a public good and you can argue it’s a public good and there’s grants for it, and there’s funding that can be done from a government perspective, it’s hard to provide a solution for this without that type of movement. But I would love to see those movements more and more.
Karen: Yeah. Yeah. What you’re describing here is someone’s trying to find a solution to address the fact that a lot of us simply don’t trust these companies with our data anymore. We tell them they can’t have our phone numbers and our email addresses. I think, in large part, it’s because we don’t trust what they do with it. So what do you think would be the most important thing that these companies would do to earn and to keep your trust, if that’s even possible? Some people say it’s not, some people say it is.
Paola: My approach, unfortunately, for it is just that I understand this is the way the world works. It sucks not to deal with it in terms of trust until they start compensating. The big ones. I don’t mind the smaller ones, because it’s really hard for them to do this. The big ones, until they start actually giving back, from that perspective, there’s no real trust that’s going to work. And I don’t think it’ll ever work out. Because most of these companies are already public and they need to have a good run at their stocks. That’s a little bit of a different approach. But from a trust and data breach and privacy perspective, I don’t think that’s ever going to work until there is a way for them to compensate people back or to be very transparent about it.
On the other side of this, when I answer this question (public distrust of AI and tech companies), what I emphasize is that right now we are still in this – people are fighting, and in the cartoons, where it’s just smoke and people, you just see the heads popping in and out. Sometimes I still think we’re in that. We’re not even at the end of that yet. There’s so many decisions being made over big bets. I think Oracle just laid off, like, 30,000 people this morning. It was a huge number of tech workers. And that’s placing a big bet on the fact that AI is able to do what needs to be done. And maybe they do need to lay off some people. But I don’t think it’s that number. It is far from that number. But like I said, we were still in that sort of fighting mode that nobody really understands if it’s the right choice or not, until maybe a year from now, even two years from now.
So there’s this sort of a sentiment there that I really don’t think one person knows what is happening, where things are heading. A lot of people can predict where things are heading and, but until we’re there, it’s really hard to see it. There’s way too many different perspectives and the technology is evolving way too fast for anyone to keep up at the moment, going from the public distrust.
So from that perspective, I think people really need to understand what AI cannot give you. And that is human connection and relationships and warm introductions that AI can’t replace. They can’t. And I think a lot of these companies know that.
That’s why I mentioned here, based off what I was doing, like Curio events, coworking events. You can get sponsorships for that. There’s tons of sponsored events around letting people come in and work with one another, but also work with the tool. So that they can understand the tool and they can better build their project, but also meet people around them. And they’re providing the tool and space to do that. I think that is the community perspective and the brand awareness. A brand evangelist perspective that they’re doing it is good for their business. But it’s also what I can imagine what people within the tech space are looking for, as well as just, “Hey, I have no clue what’s happening. Can you help me?” “Hey, I want to get a job. I think this tool is going to help it.” And then I was like, “Oh, actually I know someone who needs your skills.” Those organic interactions, you cannot get that from AI, but a lot of these companies can create or sponsor those types of events that provide those types of interactions.
Many companies, like the smaller ones – they use AI. They think it’s everything, that it’s going to replace everybody. But chances are those types of people never actually worked in every department. So someone saying, like, “Oh, you don’t need a software engineer any more because AI can code for you” – It’s kind of saying, technically yes, you could cook in your kitchen, but a Michelin chef might know how to use a potato in ten different ways, versus you with two. That’s part of the smoke that we’re in, those types of sentiments.
But at the end of the day, in terms of trust, I think AI companies that put forward the fact that, “Hey, we’re still limited. You cannot use us to replace people. You cannot use us to replace these types of relationships. You cannot use us to replace these types of talent or even professionalism.” And also, I’m just going to put this out. I work with many startups and founders and every one, a lot of my work wasn’t actually the deliverables. I still delivered. But I would say that’s not the hardest part of my job. The hardest part of my job was not building what I needed to build. The hardest part of my job was not delivering within the deadlines or not. The hardest part of my job is making sure that the egos are in check and people’s emotional regulations are – you know, they start to freak out and be like, “Oh, we’re raising so much money.” A lot of my job is saying, “No, we’re not. We’re okay. Here we go.” while still delivering. Holding that politics and holding that space for people to freak out, that’s a huge part of my job. That is not something that you can easily put on AI.
And most people are also very lazy. They don’t want to read that much output from AI. They really don’t. Like the whole book, “Don’t Make Me Think.” That’s a good rule for everyone. I used to write lots of sentences. People don’t read it. I write amazing documentation. People still want to talk to me and just ask me questions, even though the answers are already in the document.
People don’t like to think. And AI can help you there, but it just does not replace that sort of connection, that humanness of it all. That’s just what I’m going to end with. I don’t know if I answered?
Karen: No, you did: companies being upfront about what they can and can’t do and being honest about how they’re using what we give them and what we don’t – I think that’s a good summary of what it would take for you to feel like a company could be trusted.
I appreciate you making all the time for this interview! Is there anything else that you would like to share with our audience?
Paola: No, I thank you. I know that we took a long time, I appreciate your time. I would just like to emphasize: warm introductions and relationships can never be fully replaced by any Android or AI and technology. I think that we don’t have the opportunity right now, but eventually we will have that hindsight.
For me, I’d love to meet with more startup founders and builders who are using AI tools. And if they’re in Toronto or Vancouver, I host a Curio event so they can come and work with similar people here. For those who are also working in startups and building their own products, I would love to interview you and learn more about your experience and post that as well.
And I’m hosting office hours as well. As product manager, if you want any sounding board from growth management, product management, what to build, technical implementation, what you need, how to get to your next milestone, I have office hours for you. And so if you’re ever interested in any of that, I’d love to connect with you. Yeah.
Karen: Great. We’ll put your contact information into the written interview. What would be the best way for someone to get in touch?
Paola: Yeah, get in touch! My email is Paola, P-A-O-L-A, at PSProductPerson.com. You could also find me anywhere from Instagram, LinkedIn, TikTok even, Bluesky. I have Blue Sky, and my handle name is “PS Product Person”, P S P-R-O-D-U-C-T P-E-R-S-O-N. My LinkedIn is probably the best one. “Paola Santiago”, P-A-O-L-A S-A-N-T-I-A-G-O.
I do have a Substack. It is “PS Product Person”, as well. Substack is where we met. And yes, please connect with me there. I love to write articles. I don’t write as much as I would love to, but I try once or twice a month. Just things that I find interesting. There’s so many news articles out there, so I’d curate it in terms of how it might affect you. And honestly, when I have more time, I like to put a sociological perspective on everything as well.
Karen: Sounds great. Paola, thank you so much for joining me. It was really great to get to know you and get to know your work. And I’m thrilled that we had this chance to talk today. Thank you.
Paola: Yeah. Thank you, Karen.
Interview References and Links
Paola Santiago on LinkedIn
Paola Santiago on Bluesky
Paola Santiago on Substack (PS Product Person Founder Series)
About this interview series and newsletter
This post is part of our AI6P interview series on “AI, Software, and Wetware”. It showcases how real people around the world are using their wetware (brains and human intelligence) with AI-based software tools, or are being affected by AI.
And we’re all being affected by AI nowadays in our daily lives, perhaps more than we realize. For some examples, see post “But I Don’t Use AI”:
We want to hear from a diverse pool of people worldwide in a variety of roles. (No technical experience with AI is required.) If you’re interested in being a featured interview guest, anonymous or with credit, please check out our guest FAQ and get in touch!
6 'P's in AI Pods (AI6P) is a 100% human-authored, 100% reader-supported publication. (No ads, no affiliate links, no paywalls on new posts). All new posts are FREE to read and listen to. To automatically receive new AI6P posts and support our work, consider becoming a subscriber:
Series Credits and References
Disclaimer: This content is for informational purposes only and does not and should not be considered professional advice. Information is believed to be current at the time of publication but may become outdated. Please verify details before relying on it. All works, downloads, and services provided through 6 'P's in AI Pods (AI6P) publication are subject to the Publisher Terms available here. By using this content you agree to the Publisher Terms.
Audio Sound Effect from Pixabay
Microphone photo by Michal Czyz on Unsplash (contact Michal Czyz on LinkedIn)
Credit to CIPRI (Cultural Intellectual Property Rights Initiative®) for their “3Cs' Rule: Consent. Credit. Compensation©.”
Credit to Beth Spencer for the “Created With Human Intelligence” badge we use to reflect our commitment that content in these interviews will be human-created:
If you enjoyed this interview, my guest and I would love to have your support via a heart, share, restack, or Note! (One-time tips or voluntary donations via paid subscription are always welcome and appreciated, too 😊)
