#4 - Computer use is an anti-pattern for agents Ship the Diff https://shipthediff.com/episodes/004-computer-use-antipattern/ Computer use lets AI agents operate software designed for humans, but it is a slow and fragile workaround. Kasper and computerlove.tech co-founder Jonas Høgh Kyhse-Andersen explore what changes when agents become first-class users of software. YouTube: https://youtu.be/nqUbwZA08g8 Transcript 00:00 Hello everyone and welcome to Ship the Diff. I'm just checking if the mic is on. 00:05 It is. I brought my co-founder Jonas today. Welcome. 00:08 Thank you. 00:09 Yeah, to the show. Me and Jonas have a podcast together and we're actually used to speaking Danish in the podcast. 00:17 And you know, all the time basically. So it's a little weird to sit here and speak English. 00:23 But I thought I wanted to bring you just to introduce you on the podcast and maybe also just talk a little bit about the business and how it's going. 00:36 It's going well. 00:38 Yeah. 00:39 So I guess you could say that the business is not scaling well because we're only three people. 00:45 Yeah, exactly. 00:45 And we can only do so much. 00:48 Yeah. 00:50 The 0th episode of Ship the Diff is called Hours Don't Scale. 00:54 Yeah. 00:55 Hours Don't Scale. 00:56 Yeah. So everybody wants to scale their startups, right? 00:59 Yeah. 01:00 And I think that's basically one of our biggest issues right now that we don't know how to scale. 01:06 Also in terms of the software we're developing and creating, obviously we have agents, but come on. 01:12 Even though I can start up that many agents, it's really hard to, as a single person, actually be able to be productive and push a lot of features or try stuff out. 01:26 Yeah. 01:27 So, yeah. 01:28 Also, if you're scaling software to other companies, like going out and creating custom stuff, you can't do that as a single person. 01:37 You have to have some people on board, right? 01:39 Yeah, exactly. 01:40 So, it's kind of the issue we have right now. 01:43 Yeah. 01:43 Because requests are really coming in. 01:45 They really are right now at the moment and we don't know how to handle it actually right now. 01:49 So, we need to build some software that we can scale super well, right? 01:53 Yeah. 01:54 I guess. 01:55 I mean, we can just send them to our agent. 01:57 The agent can do the conversation and figure out what is it that we need to build. 02:02 Because one of the biggest issues these days when you get a request like that, people go, 02:07 oh, we need to build this. 02:09 We need to fix this issue. 02:11 We need software. 02:12 We're migrating from this to that or something like that. 02:16 It's always, the question is always, okay, so what exactly is it that we are going to solve? 02:22 What is the problem? 02:23 How do we specify what the criteria for success are? 02:27 Yeah. 02:27 How do we measure that we actually hit the target? 02:30 It's none of the code that's the issue. 02:32 It's everything else around it. 02:35 And, yeah. 02:36 But that's also maybe a potential for agents to be involved in solving some of those things, right? 02:41 So there's opportunity everywhere, to be honest. 02:44 Yeah. 02:44 It's really hard to focus on what to deliver. 02:49 We've also had a couple of customers where they wanted a series of, you know, AI things, right? 02:55 Like, I want this solution, this solution, this solution, this solution. 02:58 And then we could just basically see that, okay, you need to focus on your harness, basically. 03:02 And then, like, you know, MCP connections and instructions and skills and stuff like that. 03:09 And if you get that right, like, if you build a foundation to it, you can just, you know, delegate stuff to the agents and actually get those problems solved, right? 03:22 Also, having worked with agents as much as we have and using it in daily, you know, work and building all the integrations yourself, 03:32 okay, I need, for all the internal stuff we do at the... 03:35 Yeah, we do at the... 03:36 We do at the... 03:37 You're building a lot, you know, with Hermes, internally on Slack. 03:41 Everything. 03:41 A really good pattern that I have discovered is, like, every single time, it's basically harness engineering on a greater scale, I guess. 03:49 Every time we have something like, oh, but the agent couldn't do this or couldn't answer that or whatever. 03:55 Also, nah, it's just because it needs to have access to this system. 03:59 And you just go and build that access to it. 04:00 And straight away, it just feels magical because you have a pipeline that just pushes whatever needs to be pushed to the MCP server, 04:09 whatever change, integration that needs to be done. 04:11 And it almost happens, like, within five to ten minutes. 04:15 And then in the Slack channel afterwards, you can just go, oh, please try again. 04:19 And it just succeeds with the task that you gave it after that, which is, like, okay. 04:25 And now, in the future, the agent can solve this issue and workflow for you, right? 04:30 So you can see how quickly you can... 04:32 It's compounding, basically, right? 04:33 Iterates and sort of, like, customize your agent to do what it needs to do. 04:40 And I think also that's one of the visions that we have, sort of, like, bringing this to everybody out there. 04:47 Yeah, the agentic organization, really, right? 04:48 They can basically... 04:50 Maybe they think that they need, like, a UI. 04:54 All I need is just a new custom UI for these workers out here so I can... 04:58 And for this use case, right? 05:00 But maybe they don't even need, like, a GUI. 05:04 No, they don't need a UI, right? 05:06 They need a user interface, maybe an agent interface. 05:08 We can't call that AI. 05:09 That would be really... 05:10 But it is, like, a new agent interface that they probably need instead of a new graphical user interface. 05:19 Yeah. 05:20 And that's just so weird and interesting. 05:22 People need to get used to that software. 05:24 You can have an API that you expose to the agent that is pretty, like, foundational. 05:29 And then, you know, in the old days, you would build a UI around that and actually... 05:34 Like, oh, I want a button there so I can do this. 05:37 There's no need for that anymore. 05:38 You can just, you know, say what you want and actually have the agent do it with the primitives it has available, right? 05:44 And actually, one of... 05:45 It becomes... 05:46 The graphical user interface becomes, like, a friction point and an obstacle for the agent more than it... 05:54 It's, like... 05:56 That's why we have computer use from, like, OpenAI because, okay, they know that the world is not going to change and be agentic first, 06:05 like, when we have had, like, the internet came along and we needed to expose, like, backends through APIs so we can, sort of, like, make everything work together and so on. 06:18 Now, we have the same problem, just in agent world, right? 06:22 So, we need MCPs everywhere for the agents to be able to do stuff in this SaaS solution or this business or whatever. 06:31 But we're not there yet. 06:33 And some software is probably too complicated. 06:37 And so, they train the models for computer use, right? 06:41 Because, okay, then the capability of the model and the agent will just, like, be really high. 06:47 Do you think computer use is the future? 06:49 Or... 06:49 I think computer use is a solution to a problem that doesn't follow first principles. 06:57 Yeah. 06:57 Because first principles would say that you don't need the GUI, right? 07:03 You don't... 07:03 That's the, like, waste... 07:06 Clearly, clearly. 07:06 That's a wasted interface because your interface with the agent is you talking... 07:11 For some tasks, right? 07:13 For some tasks. 07:14 For some applications. 07:14 Yeah, visual stuff, obviously, like, graphical interactions. 07:19 You need some kind of, like, GUI to be able to see it and obviously, probably, you also... 07:26 If you're building a 3D model, you want to be able to, with your hands, do something. 07:31 It's very inefficient to tell the agent to scroll the model five degrees and then do five degrees again, you know? 07:39 Yeah. 07:39 So, there's, like, some things that you don't want an agent to do for you. 07:45 You want the real world to, sort of, like, interact with that. 07:49 And those interfaces are still going to be around, but... 07:53 So, is computer use just a hack for, you know, applying agents for a current setup? 07:58 Yeah, for some applications, like, for instance, go to this website and order a pizza for me by going into the browser, pressing it, and so... 08:09 That is just completely useless. 08:13 That's inefficient. 08:13 That's so inefficient and really not the way you want to do it. 08:16 You want, obviously, an MCP solution that can do that for you, where it is extremely fast for the agent to do the thing and not get back to you after, like, two minutes or whatever it takes to spin up that browser and so on. 08:33 Obviously, there's, like, headless browsers that make it faster and so on and so on. 08:37 But still, you... 08:38 And that, obviously, isn't a GUI, but it's still, like, a browser that is clicking on buttons behind the scenes. 08:44 So, I think that there's a lot of wasted or there's a lot of things that are going to be obsolete if and when the world moves into a more agentic, like, interface, language-based application interface. 09:03 I don't know what to call it yet. 09:05 Agent interface, I guess. 09:06 Yeah. 09:07 Language interface is also... 09:08 Could be... 09:08 Maybe it's something that's more generic in terms of... 09:12 Maybe we don't call it agents tomorrow, but some kind of, like, semantic interface. 09:19 Yeah. 09:19 That, I think, is... 09:23 Yeah. 09:23 Cool. 09:25 A lot of ranting. 09:27 So, should... 09:28 Like, what's your thoughts on, you know, we're developing a product, right? 09:31 We are launching soon. 09:33 Yeah. 09:34 Within a couple weeks, I guess, it's planned. 09:37 Could we tease a little bit about it? 09:38 I have talked about it sometimes before on the podcast, but... 09:41 I mean... 09:42 I bought the domains, right? 09:44 You bought the domains. 09:45 So, you know, nobody can... 09:46 The teaser is a little bit of what I actually said, I think, because the vision is what we're 09:53 basically talking about here. 09:55 But how to deliver this interface to agents and systems out there existing already is the 10:03 big issue, right? 10:04 And big companies also. 10:05 Yeah. 10:05 So... 10:07 Securely. 10:08 You could say that, like, computer use is the anti-pattern to what we're trying to 10:12 do. 10:14 Trying to make agents succeed in all kinds of different systems. 10:18 Obviously, you can do that through computer use if you have the GUI and so on and so on. 10:22 But there's an issue there, right, with computer use because on behalf of who is that agent doing 10:28 it and is that actually secure? 10:30 The user has access to delete stuff and you can accidentally press the delete button. 10:35 And how... 10:36 Because it's like a graphical user interface, how do you sort of like control that permission? 10:44 It's really not that easy to solve those security issues when you have like computer use stuff. 10:52 So, I think our approach is more leaning into the standards of using MCP, the protocol. 11:00 And sort of like following along there and just basically enabling you to highly customize 11:07 and extend a service that is exposed over MCP to be able to talk, to understand and... 11:19 To give employees that magical feeling when it just does something, right? 11:24 Yeah. 11:24 Because it's connected and it has a closed feedback loop. 11:26 So, but not just, you know, taking like... 11:32 There's a lot of MCP solutions out there that does this connector thing behind the scenes to 11:38 connect to this system, connect to this system. 11:40 And they all ship with like a thousand plus connectors for all the systems and very generic. 11:47 But they don't necessarily capture the business workflow that you as a business owner, you want. 11:56 You might actually have a workflow where you use two systems or three systems and maybe 12:04 one or two of those systems you couldn't expect anyone to have an out of the box connector to 12:09 because it's your system, you built it. 12:11 Yeah. 12:11 It's over here running with an API that's proprietary. 12:14 Only you understand it's your business domain. 12:16 But how do you get up to speed quickly connecting those things if that's the case, right? 12:22 That's the problem we're trying to solve. 12:26 So, a highly extendable MCP solution that an agent could actually build the integrations for you. 12:33 So you can... 12:34 A bit meta, I guess. 12:35 A little bit of... 12:35 I'm thinking maybe to call it the... 12:38 I call it if the people out there know this... 12:41 the Pi coding agent. 12:43 Yeah. 12:43 Maybe it's getting too technical now, but... 12:46 It's not, it's not. 12:47 It's an extendable like coding agent harness. 12:51 This is sort of like an MCP that's extendable. 12:55 Yeah. 12:55 Like by an agent itself, right? 12:59 So yeah, you could call it that. 13:01 Great. 13:01 We've been talking to... 13:05 We've been talking in 13 minutes right now. 13:07 And I think that's enough. 13:08 Not seconds. 13:08 That's enough for today. 13:11 Yeah. 13:11 Thank you so much for joining. 13:13 No problem. 13:13 So, but thank you so much for following. 13:17 Look forward to talking to you again next week. 13:19 Bye.