Can AI Tell You What Your Goals Are? (Ep. 163)

Welcome to Never Rewrite. I'm Isaac Ascu, Dustin Ray, and I'm Jeffrey Sherman. And today we're going to ponder whether AI can tell you what your goals are. So this we were kicking around before we hit record.

A thing that is happening a lot uh especially when you have less technical people coding is they say okay I' I've I've got I told AI to do something and it's produced some code and I don't know how to test this it's like well what is it supposed to do like what goal what problem are you trying to solve and that tells you how you would test like that that's the workflow that you would test and if you can't you if you if you don't inherently know immediately how you would actually test this. You probably don't know what you asked it to do.

I also think that, you know, not to be too diving into the AI realm, but you could also just ask the AI, how do I test this? Like if you're that stuck, you really can just say, can you bootstrap this thing for me and boot it locally and give me a link and let me poke around with it. you know, um I would be concerned if somebody was able to be savvy enough to generate the code they think solves the problem, but not savvy enough to load it to see if it does the thing they want it to do.

I was thinking about it more in terms of soggy requirements and people like, oh well, it wrote a bunch of code. I asked it to do something and it did something and but I don't know if it did what I really wanted because I don't really know what I wanted, right? Um, yeah, Dustin, you've been working with a lot of founders. How how have you seen this playing out? Yeah, I think it sounds simple like when we're like talking about it and I think even for engineers, like even for me, it sounds simple until I go and sit down and try to build it.

And I think everyone even vibe coders will have had this experience where they feel like they explained it very well and then the result that they got on screen like wasn't what they expected. That gap right there is what we're talking about, right? like the soggy requirements is what leads to that in my experience. So like even things that I think are just so super obvious that I don't need to explain to the AI again unless I have some enforcement in place for things such as even as simple as like use the same logo through the entire platform.

Like don't just generate your own S SVG version of our logo like in place of using the asset that exists in the codebase. Like always just use that asset. You would think that's obvious, but I'm guessing that happened to you where it has happened multiple times that I can't believe that I have to keep like I have to like now put something in my my agent MD to like stop it from doing that. So, it like always looks for the brand asset. So, I'm working on like a brand scale kind of thing where like it looks for these things automatically.

But yeah, it's a problem I've run into enough times that like I've I'm now reaching for building something that does it automatically [laughter] if that tells you anything. Uh I this morning I was coding and it was like oh well I need to set save the current state and then run this and then set the new state and then restore the old state. I'm like that no I I set the like the pro it's a it's a single task process. I it already bootstraps the state that we want and then the process dies. You don't need to save state and restore it. just go.

Uh, but yeah, like that was I I was looking at this thing and I was like twothirds of the code that it wrote was all about saving and restoring state and I'm like why did you do that? But it was obvious to me because I knew what my goal was and my goal was not. I guess maybe part of what we're talking about, we're thinking is AI encourages you to take giant steps and giant steps are inherently dangerous in coding. You want many more much smaller steps, as small a step as possible. Yeah.

I think there's also like this feedback loop where like it'll it'll get you excited especially if you're not technical.

It'll get you excited about what it can do and it'll overproduce like you know imagine the guy who comes to you and like hey I have an app idea that used to be the dreaded fear of every engineer back in the day right now you can say okay you got the app idea go over there and tell it to chat GBT and it'll generate this you know just bits of code and stuff you didn't even ask for that it thinks oh this is what an app usually would have and it just generates all kinds of things and maybe one of those things loosely solves the problem if there even was a problem but the idea or the fun thing that person wanted to build, but then it's got all these other kind of cool things and bells and whistles that it generated along with it.

They're like, "Whoa, this is done even more than I thought it could do." And so people get lost. Uh they don't have the requirements. AI just comes up with things producty people have historically come up with to monetize things and throws them in there with it and they go, "That's great. Yeah, yeah, yeah." And like I'm they're almost asking it or begging the AI to come up with an idea. Come up with everything for me. I want I want you to come up with the idea.

I want you to execute on it and I want you to test it and I want you to just make me money. Go make me money. Yeah. I think it's it's easy too to point to like the AI, right? And like especially because that's what everybody's using right now. But I would even argue that those problems existed before, right? Like it just was slower. So it's the fact that I think it's so in the mind of everyone is like now everybody can experience it.

It's not just like engineers talking to engineers about like look at these requirements like how am I supposed to know what they mean when they say add notifications like we're talking push notifications should I add it in the email is that going to SMS like there's so many additional requirements that like go into even something that sounds simple right I think the same thing we're just seeing that at fast pace with the AI and it's like if you don't understand what you're trying to achieve and build even if it's on the behalf of your users like I I think you're going to have a frustrating experience both with humans and with AI.

The only reason that I think humans improve the experience is because they're the buffer between you and the AI that are asking you the right questions to then get into the AI and the build part of it. Right? Before that might have been a BA or a project manager or a product manager, you know, now it's, you know, it's in between. It's all mixed up. Yeah. I'm reminded of there's an old saying that the work will expand to fill the allow uh the available time and sort of AI is giving us the feature version of that.

The features will expand to fill your token window. You you were saying like it's it will generate all this stuff that nobody asks for, right? It's solving problems that nobody's asking for and you it becomes a system and then you actually put this thing out because now it's generated all this stuff. You put it out in front of a user and maybe it solved the user's problem, maybe it didn't, but you have all this other stuff and it creates all this.

It's not whip at that point because it's in production theoretically, but it's stuff that now has to be maintained and used that maybe nobody cared about and nobody wanted in the first place. Mhm. And it's not as bad as the old days where you spent six weeks of developer time making a feature that nobody cared about. Nobody but still it's there and it's a distraction. Like it costs money, token money. Well, it token money, but I was thinking more of if you've got 30 features. If you build an app and uh my son was using a cool app.

It's it's a Shazam for birds. you you whip out your phone and you hit listen and it tells you what birds are singing in the background. Very cool. Right?

If you if if you know if you had built that thing and it had also added a I don't know add new bird right I've got an unknown bird which you know is not really a thing at least in North America right there are the the the set of birds and that you're going to hear in outside of Chicago is a solved set right you don't need you don't need functionality but like it could build it and then now it would have to have right?

All of these things about uploading samples and managing your buckets, you know, your S3 buckets or whatever you like there's a whole feature set and a whole lot of code for something that's not a real problem, right? Yeah. I'm almost wondering if you know maybe we change our agentic flows to having a folder that's like a customer a mock customer like this is John and John uses the business. the whole reason he even cares about what you're building is this is his problem.

You just like a bunch of personas in a folder um or or problems in a folder, you know, call it what you will. But the idea being like every time you try to generate any new feature or change an existing one, it has to look back at that folder and make sure that there is no regression on existing problems that no are no longer solved or that the new ones that exist in the folder have been solved. And everything just revolves around those personas because it like forces it to be customer first and not just like do random Well, go ahead, Dustin.

No, go ahead. Uh, I was thinking a little stronger of tell you you're telling the agent to solve a problem and then tell it to stop after it solves a problem. Don't have it go on and be like, oh, you know, it's like the Amazon recommendation. customers who solve these problems also like to solve these other problems. Like, no, no, no, don't.

Part of the persona set is John loves his main problem solved and he hates any extra features [laughter] that distract him and confuse him when trying to get to his main features that Yes, I think simplicity is a real feature. That is a real feature. I think you can definitely call that out. I think that there's there's something maybe in between what you guys are are saying or maybe you're even talking past each other a little bit.

what I'm what I'm hearing is I think more aligned with what what what Isaac is saying that what I've seen in practice is and what I'm using you know personally I don't want to like talk about it here but I'm publishing something very soon that's open source that like helps solve this problem that I'm actively experiencing um but I I am of the same mind that you need something that actually isn't AI that tracks what you want your platform to be both in the human sense.

So there like as a human I can read I call them requirements or specs right like the these different you know ideas PRDS right like there's this concept of there's some some layer that defines your product and I think Isaac that's kind of like what you're getting towards is there's some durable layer that both humans and the agents can access that defines customer personas or things about the product or the domain that it can always access. It's not based on like the model's memory.

it's in the codebase or in a special repository or something like that, right? Uh and I think the solution to the problem that that I currently am using that I think is pretty effective is assigning ids just like we do with you know Jira tickets or linear issues to the requirements themselves and then tracking that across the different code bases. So the different repositories that make up the platform because it's going to involve your AI or your API, your your front end, maybe your mobile app needs to pull from that.

Maybe you have a customer portal that also needs to pull from that API. So like when you get a change to a feature that spans all of those things, how and those are going to be different agents or potentially even different humans running different agents in those repositories. How do you get a coherent feature across that platform? that that to me is the challenge that I don't think that the AI itself LLMs do not solve that problem. The what I think solves that problem is a deterministic engine that holds on to those ids like a database.

And then so if the agent tries to switch those requirements out without some type of approval process, like it just deletes a requirement, which if you've worked with AI enough, you've probably seen it just steamroll things you've built and you're like, "No, we still have to have that in there. You can't just delete that to make your thing work." But you know, my bad for not including that on the requirements, right? So I think that is what stops it. Uh and I I've had some success with this.

Uh and that's why I'm trying to publish a tool about it. So I'll have more to come on. Yeah, it sounds almost like an editorial process and having an editor. Uh because feature coherence across multiple repositories that was hard, you know, even in the pre correct LLM days like that's not a like I have built many features that were totally incoherent. Yeah, because the requirements got mangled across teams. Yeah, the work for this solution has been from years before we even had, you know, coding agents.

You know, this has been a problem that exists, especially if you've worked in in agencies or or places where you're working for clients and you have requirement sets and you're trying to like get legal around a set of scope like this is a very difficult thing to then prove in the deliverable, right? So, and I think the same goes for like product teams as well. they have the same burden of proof.

Uh but agencies and like freelancers directly like run into this frequently because they're seeing the same thing at every client and it's like well how do I keep the requirements true through the entire platform even working with their different teams or different product managers or different engineers or different agents. Uh you know I think that's a this is a problem that's existed in softwares forever. Yeah, right. be like be secure always or don't leak PII, right? Like that that's a thing.

And I mean I have had so many insecurities and so many so many security issues and so much PII leaking. But yeah. Yeah. Multi-tenant like isolation principles. This is something I've found is like something you have to have like baked in. You can't like depend on the LLM to respect tenant boundaries.

like it it has to be baked into the services like into the rules like all the way through like it can't be dependent on its judgment right and I think and to be fair I think the same is true for humans and I've been saying that for years like we shouldn't be depending on humans to like remember to pass a tenant ID every request like it should be baked into the services that they use right you've got to make it hard to do the wrong thing right right make it really hard yeah make it really hard to do the wrong thing and I think on the same topic right the same thing of like how do I know that the agent is doing something secure or how do I make sure it's not crossing boundaries with multiple clients at once well don't give it multiple client accesses at the same time it it has to be contained like it's zero trust so I think the same goes for your testing and your security like apply the same principles as if it's just a junior employee or just someone that you just don't trust and you will be fine like that will work you know yeah so going back to the topic at And can the agent tell you what your goal is?

[snorts] Not yet. But if you ask the agent, yeah, if you ask the agent for more goals, is that a good sign? Like uh or if you don't know how to test it or going back uh you know, if if you say, "This is my goal. I'm going to build an app." And you then say, "Okay, I've got this app. Now what can it I think that the right question to ask is is help me articulate the idea like show me the gaps.

A thing I add to all my prompts is like point out any ambiguities and then ask me about them instead of assuming and if you make any assumptions like you know best judgments call that out so that I can when I look through the log I can be like okay it made a choice here that was wrong and now I can train against that choice.

So I think it's the same thing you it's a double feedback loop like the same you want it to be able to train you and do research on your behalf you want to then feed that back into it to get a better output out of your prompt. So you're always debugging to use a technical term the prompt or the agent and not like the output itself. So and that goes for you too like I think if you are getting an output that you are not happy with that should be a feedback to you that you're missing something in your input. Yeah. Right. Yeah.

I I find I I do a lot of, you know, write this thing like write down what you're going to do and then I will read it and edit it and then I will tell you to do it and then I will be able to you'll like it's going to be much closer because we've already discussed all the ambiguity. Yeah. And by discussed I mean I you were wrong and I changed your mind because you're a machine. Yeah. Yeah. I I love it.

I've been using since the very what we were calling boomer AI like to for like architecture documents like hey I have this halfbaked idea give me examples of other teams that are doing this give me counterproofs of like why we shouldn't do it like who is feedback online about why they've done it and why you know what's better than this idea you know what are the current best practices and I think that it's like an education loop so you're educating yourself and at the same time improving the idea that you have but if you don't have that goal in mind, you could be going in the wrong direction, right?

So, like if if AI is telling you where to go, I think you will find that you're going in circles. You're not you're not you have to direct, you know, right? Well, it gets back to the fundamental truth of you need to get it in the hands of the users and you need their feedback. You don't need the AI's feedback. You need user feedback and you need to solve Yeah. You need to solve user problems, not the problems that the AI thinks you need to solve. Right. Yeah.

There was something we said before the call or before the recording I want to make sure I guess gets into this as well. And I think it was the signal, right? The signal is if you feel yourself asking the AI, what is the goal? What else could I build? What else could I add? That's probably a signal that you need to be talking more to customers or selling more or getting out there more. Like that's probably the gap, right?

If you're I think so and programmers get into this loop, you know, in the old before trying to say it's I' I've built this amazing golden hammer. What else can I build can I hit with this hammer? AUD teams go off for six months building things because they've got this amazing framework or tool or whatever that nobody cares about. Yep. Yeah. Don't think that other people's problems. Yeah. Sorry, Isaac. Go ahead. Oh, I was saying don't think your problems are other people's problems.

Um because everyone's got the app idea too and I had the app idea. I built something that solved my problem. Shared it with my friends. Some friends are excited about it and some friends are like that's not a problem I've ever had in my life. [laughter] And if I'm excited that my problem solved, it doesn't mean it's marketable or that it's it can scale and there's enough people who pay for it. It might just be something I keep to myself and it solves my problem. But customer feedback is the main key. Need people there. Yeah.

One thing that AI can do is let you solve your problems and not need it to be a a big enough problem that there could a company could exist, right? Which is or even become your competitive advantage on why you can do something that a competitor can't, right? Yes. If if you're solving problems for your professional life, right? And that's always something that the programmers could do of oh well, no, it's right.

you you can't hire a pro I mean you could but it wouldn't be cost- effective to hire a programmer to solve this problem but as a programmer I could solve it myself for myself right well thank you all right thank you all for listening I'm Jeffrey Sherman Dustin Ray Isaac Ascu and this is never rewrite