How AI Agents Will Turn Freelancers Into Orchestrators With UpWork
27:37 Share

From privacy concerns to limitless potential, AI is rapidly impacting our evolving society. In this new season of the Brave Technologist podcast, we're demystifying artificial intelligence, challenging the status quo, and empowering everyday people to embrace the digital revolution. I'm your host, Luke Malks, VP of Business Operations at Brave Software, makers of the privacy-respecting Brave browser and search engine, now powering AI with the Brave Search API. ♪

You're listening to a new episode of The Brave Technologist, and this one features Andrew Rabinovich, who is the VP, Head of AI and Machine Learning at Upwork, and a prominent figure in the field of computer vision and machine learning. He's known for his work in deep learning, particularly in applications such as object recognition, scene understanding, augmented reality, and multitask learning. In 2020, he founded Headroom, an AI video collaboration platform, which was acquired in 2023. Currently, he leads AI and engineering at Upwork.

In this episode, we discussed job security in the age of AI and ways the freelance economy is already responding to its opportunities and challenges, ways that AI agents will turn workers into managers, managing the machines, human-centered AI, and the type of solutions Upwork is building to empower their community amidst AI's uncertainties. And now for this week's episode of The Brave Technologist. Andrew, welcome to The Brave Technologist. How are you doing today? Excellent. Happy to be here, Luke.

Yeah, yeah. Thanks for joining. I'm super interested in this conversation. There's got to be a lot of interesting things going on at Upwork now. What's the most exciting project you're working on right now that you can share with our listeners?

Sure. So I joined Upwork about a year and a few months ago when my startup headroom that was working on video conferencing was acquired. And the main reason for choosing Upwork was because in the last probably 10 years now, I've been working on this concept of human-centered engagement.

AI as opposed to general artificial intelligence. And as you know, Upwork is the world's largest freelancer marketplace, which I find to be the essential components to achieving AGI through this human-machine integration. Awesome. How do you see AI transforming the freelance economy? What kind of opportunities are there and challenges for independent professionals?

We can step back and think about sort of step function changes in evolution in general, starting from steam engines and electricity and things like that. And if you're a farmer and you do everything by hand with the invention of electricity, it's not that people stop farming. They

They just start farming better, more efficiently and effectively, right? So we think that the same thing is going to happen in all digital work, which is what Upwork helps to facilitate. And that is that freelancers will be able to do much more, much more complex work with partnerships with AI. It's not that a freelancer gets replaced by a machine. It's that a freelancer with a machine can achieve much more.

Got it, got it. You're kind of touching on a good point I was going to come around to with job security. And I mean, obviously Upwork is engaged with a lot of freelancers. Like, are there ways that they're using AI with the platform now that are helping to make their efforts more productive or anything you can share on that front? There are two components that relate to your question. First is, are there tools on Upwork that help freelancers

Find work. And the second question is, are there tools on Upwork that help freelancers collaborate with AI from external sources? And the answer to the first question is yes. There is a meta agent inside of Upwork today that's called UMA, that stands for Upwork's Mindful AI, that helps both clients and freelancers collaborate.

Either find people to help do the work or for the freelancers to help them find the work. Now, to the second question, we're actively working on it and I can't talk too much about it in the open. We're working on allowing freelancers to interact with agents. And there's like 2025 is the year of agents.

so to speak. A lot of people define agents in very different ways, but irrespective of the definition, a interaction between a human and a machine is taking place today very actively. The simplest form of it is prompting. You go to chat GPT, you ask it questions, it gives you answers. That's an interaction between a human and a machine, albeit fairly primitive. So if you assume all possible things that can be done on Upwork,

From design, coding, writing, music generation, search engine optimization, like, you know, podcasting, like anything you want to do. To all the possible agents that are going to be available in the world. We think about what is the platform level interface that allows all these freelancers at Upwork to interact with all these up and coming agents that are about to help. So that doesn't exist on the platform to have, but there's more to come there.

No, it's awesome. Yeah, it's really cool to hear kind of how you guys are mindset, you know, around this and how you guys are looking at it. Can we drill down a little bit into this concept of human centric AI? Yeah. So it's a fairly loosely defined term, but a few folks, including you,

Jan LeCun and Fei-Fei Li at the Human AI Institute at Stanford talk about this. And the idea is not to replace people, but to amplify them, right? We all want to be superhuman in some regard, and this will allow us to do that. A simple analogy is we've built AI to beat humans at chess, and now at Go, right? So the question is, you don't want to play against the computer anymore because the computer will always win.

But is it possible for you to play chess with a computer against another human with a computer? Arguably, it becomes a different game. Maybe it's not called chess anymore, but it's still something net new, right? So same thing here. We want to build AI that doesn't... Like building AI in the realm of...

outside of humans is a bit awkward because we don't know what objectives to give it, right? Like if on earth we want to say, help us solve longevity, hunger strikes and lack of resources for the whole world. These are very concrete tasks that we can give to the AI. But these are things that involve humans and empower humans and better humans, right? If we talk about things

that happen on the moon that we have nothing, no idea about, then that would potentially be general intelligence, but not related to humans. So I think the things we talk about are how do we build artificial intelligence that works in the realm of human beings? In the same way, why is it that when we build humanoid robots, why do they look like people, Paul?

Because the world that we build these robots for is built for people. Therefore, these robots must resemble some kinematics and other capabilities that are consistent with humans so that they fit into the frame of reference that they're built for. Right? So this whole story started with this movie called Her. I'm sure you've seen it. And the goal was to really create...

not an assistant that would take tasks or directives from people, but a companion, someone who would be level-footed with a human and to recognize that humans are very good at certain things. We're good at emotions. We're good at intuitions. We're good at feelings. Machines, on the other hand, are

are also very good at things, but are fairly orthogonal to what I just mentioned, right? They can work forever. They have no issues memorizing everything. They have universal knowledge, things like that, but they don't have an ability to infer or have built intuitions or build relationships that are more on the emotional level than purely information gain, right? So if you combine the two together,

then you're able to really get to the next level of intelligence

call it general intelligence or not, like the tags are irrelevant. But the net new capabilities of human-machine symbiosis is next level in my understanding. Interesting. What do you think the biggest impact of this AGI is going to be? So the biggest issue today with AI in general is that it's not agentic, meaning that it doesn't have

an ability to decide what to do,

nor does it have an ability to decide when the job is completed adequately. So all the mechanics of things, we can say, go figure out where to drill the next oil hole or go figure out how to build the next rocket. Like these kinds of things machines can do and they'll get only better. I'm very confident of that. But figuring out what to work on and figuring out when has...

the solution then delivered to the desired results, that's not obvious. So understanding, like there's a lot of what we call in sort of the human world, a lot of institutional knowledge when it comes to work or relationships and so on. But there's also a lot of institutional knowledge that's not written down. And machines learn from data.

And if there is no data or if the data is not at super high volumes, then learning becomes very, very difficult. And that's why we have things like human-on-the-loop reinforcement learning approaches, right? Asking people for answers every single time is possible, but it proves to be very expensive and

and inefficient. Yet there's this sort of insight and wisdom that humans possess that I don't think machines will have in the near term. So an ability to solve very practical problems will obviously come around. But then there's also these complex social constructs that will require this pairing of humans and machines. No, it makes sense. It makes sense. Upwork seems like

a really interesting kind of arena for this given, you know, freelancers are both kind of, I would imagine they'd use you guys for like, you know, not only like kind of acquiring work, but also doing the work. Right. So you can kind of have that kind of feedback, Luke, I would imagine.

Are there areas that you're seeing the AI use really take off on the freelance side that are interesting, or is it just across the board right now? Because I imagine you're getting demand everywhere kind of thing. That's a great question, and this very much depends on the quality of the freelancer.

Take software engineering, for example. There are things like Cursor, Copilot, Reflection. There's a ton of these new tools that write code for you. I use them all the time and I write like 10 times more code than I have in the last 10 years, probably. Wow.

If you're a very advanced software engineer, you will learn how to use these tools and some do. But there are still some freelancers who believe that, you know, you open a blank terminal with VI and you just start writing code from scratch and everything is manual. Can they produce best possible code? Possibly. Will it take forever? Yes. Will it be very expensive? Yes. So what we're starting to see now is that there is a very...

soft, but a bifurcation between those who have adopted tools and know how to use them and those that are still doing things in the classical way, if you will. Interesting. As a customer or user, right, there's probably a whole bunch of ways that you can start to quantify these advancements too. And, you know, hey, you're saving this much time or you've gotten this much of an increase. Are you guys working on metrics around that? Oh, yeah. Yeah, yeah, yeah. Absolutely. So you want to measure things across

Three domains. Number one is obviously quality, right? Because when clients come in, they hope and they request a certain amount of quality. Although the way work is structured today, we are mainly responsible for finding a match. But whether that match works out or not, that happens off the platform. So the work happens off the platform. After quality, there's obviously cost because everybody wants the best stuff but cheaper. And then there's time.

And there are very interesting metrics around how you can achieve the same quality in a much, much shorter amount of time. Hence, the price goes down dramatically. And a good example for that is, for example, machine translation. If you want to

translate from French to English, you go to Upwork and you ask the search engine to find you a translator who speaks both and they can do the work. And you can start seeing that there are freelancers who are like, okay, I'll take the work for you. And in the background, they go use ChatGPT to do the translation and then they just edit and then they're able to return the work order magnitude faster than those that could do this by hand.

How is it like measuring quality too? Are there pure metrics you can get from that or is it like there's still a subjective like element where you guys are kind of making assumptions on your side? I'm just really curious. Sure. That's, that's a great question. And the answer to your question lies from the news that came from deep seek. Ironically, there are tasks that are deterministic. For example,

code completion or compilation, right? You don't need a subjective metric to determine if the written code compiles or not. You just compile it. And if it compiles, then it works. And if it doesn't, then it doesn't, right? So what DeepSeq did is they said, presumably we don't have a lot of compute resources because we're not a US-based company. We need to do this fine-tuning or post-training process that usually requires a lot of

modeling of models, evaluating models, like LLM as a judge type thing. But those models are very, very expensive to train, the judge models. And it's expensive to run them.

So once we have a proposal from the model that we're training, rather than having a LLM as a judge to evaluate the results, let's just make up a bunch of rules that are deterministic. And those take constant amount of time to run, hence cheap. And then for a lot of the tasks that we want to fine tune our models on, we'll just use that and assume that that works, right? So same thing applies here. When it comes to any generative model,

non-deterministic tasks, things are very, very subjective. And aside from getting human expert consensus, it's very difficult to derive any kind of heuristics. I mean, it's very difficult to derive metrics that are not heuristics, right? Because everybody can just say this is subjective and that's subjective. Fortunately, a lot of the work in

work domain is comfortable with that because it's been like this all along. Even like in fairly deterministic tasks like writing code, people have different coding styles, right? And if we, as we expand into this human machine cooperation, a client can come in and say, I need, I

I want this type of code written. Look at my code base and mimic the coding style that I have. So then the machine can do this. And if the exposure of human and machine is front and center, then it's very easy structurally to sort of propagate coding styles all the way to the agent and then have the freelancer evaluate it and say, yes, this is written according to the coding styles of Google or whatever. And then it moves forward. But if you do it

In an adverse case where you just give the work to a freelancer, the freelancer may look at the code that you provided as an example and write something that they're used to doing. And it's very difficult to get them to change it because unlike a machine, like the humans been trained to write code, writes code in a certain way. And it's very hard to get them to change it.

Well, it makes sense. It's really interesting. Do you all work closely with like testers or freelancers that are kind of like, you know, in like a beta or test cohort that can give you feedback on these things? And what are they excited about? We are in the process of doing that. And there are freelancers who are eager to adopt new technologies because they see it as an opportunity to get more work.

- Right. - Right? And the way I think about it is that as agents become more and more capable, you can think of the amount of work that exists today on Upwork can grow exponentially more because not only would you be able to solve much, much more complex things, you can come to Upwork and say, "Build me an open source version of Facebook." - Wow. - Now, if you say that on Upwork today,

A, you will never be able to probably find a single software engineer who would sign up for this because this is crazy, right? And even if they did, it would take like millions of dollars and years of life, right? Sure. Yeah.

If you're able to find human-machine collaborations, then I can imagine this being done in 30 days for 100,000 bucks. That's on the complex side of things. But on the simple side of things, we can start thinking about agents being able to solve almost these microtasks with insane frequency.

Right. As a business or as a consumer, you can go to Upwork and you can just say, I want this and I want that. And every project would be like 10, 50 bucks. But there will just be so many that a given freelancer will be able to manage the whole scope.

of hundreds of little projects that they may do with different agents, therefore increasing the demand for their time. Yeah, I was going to ask. I rarely hear people say things like insane frequency. So imagine somebody doing tens of projects can scale up to hundreds. Is that kind of... Yes. So the interesting bit is that their direct involvement or time spent on each of those projects will go down because then they will be in this mode of...

managing slash verifying or guiding as opposed to doing, you know, and you can sort of see this with people as well, right? Like, let's say you are a software engineer and then you're an individual contributor. You're responsible for, you know, writing the design documents to actually coding, to testing, to deployment, whatever. Then you look at someone who manages a thousand person organization. They don't do any of it.

but they make sure that everybody gets their stuff done. So the same type of orchestration, if you will, will happen between humans and machines. I tell there are machines that become more advanced than humans, but I think we can leave that conversation for the next dialogue. Yeah, so going from herding cats to herding agents kind of scenario, right? Correct. Yeah, and these agents...

They're very interesting because they don't have any kind of social issues. They don't have any kind of preferences. They don't have any opinions. They just like do things, right? So then it becomes much easier for you as the orchestrator to get really efficient at it. So long that there is the right abstraction layers, if you will, to connect them and to make the interfaces efficient. That's awesome. I'm wondering too,

It sounds like this to be helpful in project management and even the direct work itself. What about on the business side? Do you see agents helping with contract negotiations and things like that? Agent-to-agent negotiation types of things in the future? Or is there just a human part of this that's just always going to be there? So with humans, it's literally a marketplace, right? So the way things work today is that a client comes on the platform, meets with Uma, tells Uma what they want.

describing the project. UMA puts together a project plan. Then it uses that project plan and the search engine to find a set of possible candidates, freelancers. Then it helps those freelancers write job proposals on their behalf to the client. Then UMA switches the hats and helps the client pick the right candidate. And then...

the job sort of starts. And then the UMA will facilitate any kind of negotiations and so forth. With agents, it's trivial, right? There's like per token price and there's nothing to negotiate, right? So long that you have the right interfaces, APIs for lack of a better word, then everything is super seamless. You can forecast

ahead of time how much things will cost because it's fixed, right? And there's always going to be a competition where, as you see it is today, a new model comes out and they're like, the price per token is a cent cheaper or the context window is 10x bigger, you know? And just through this natural competition, you get this evolution in quality and efficiency. Wow. Yeah.

Wow. That's awesome. I mean, it sounds like pretty comprehensive now. Like it's interesting to see kind of imagine where this will go. It's getting there. You know, it's not an overnight change. We have to remember that Upwork has been around for 20 years. You know, so on one hand, internally, we build a lot of tools and innovations and tech.

But on the other hand, back to the human component, there's a lot of legacy momentum. By inertia, people do certain things a certain way. And despite having all the latest and greatest tools available, they still need time to adjust to them and to adapt.

forego all the sort of things that they have verified in the past that work or not. Sure, sure. Keep culture ethos kind of getting that fit within the org. Yeah, that makes sense. I mean, and that's one of those things that's interesting having 20 years of just the data and experience and maturity in the market and then being able to apply these tools in that context compared to like, you know, being a freelancer or startup.

or whatever. Like it's super interesting. I'm sure a lot of freelancers, those that haven't caught the bug yet with this, but are interested, is there anything you recommend or for them getting into kind of getting prepared to get into this AI driven job market or mindset or tips or anything like that? I would recommend figuring out which parts of their own work they can outsource to the agents. Yeah.

And then start looking at much more complicated tasks that they themselves can solve with the presence of an agent. So if you used to build websites in Tailwind or whatever, React Native or whatever, start thinking about building much more dynamic websites.

web applications that incorporate user interactions, search engine optimizations, all kinds of things. So you can actually tackle much larger projects as opposed to just doing this one thing because you rely only on yourself. Interesting. That's awesome. And I know we covered a lot today. Is there anything we didn't cover that you want our audience to know about? I think we did speak about a lot of things. One thing

that I'm particularly excited about is there needs to exist an interface between humans and machines that allows us to

interact in the most efficient ways, because if we don't do that, then it won't work. There are startups and larger companies that are thinking about this. And I think this will be the next sort of missing piece of this AGI puzzle, if you will. So the machines are getting smarter. The need for them to interact with expert humans is

Just like you heard from OpenAI that now you can have a PhD level chat GPT or a master's level chat GPT. So clearly, the expertise of people go beyond having an internet connection and a pulse. People's knowledge is really valuable.

That continues to grow. People go to colleges, although, you know, it's questionable these days. But the interface between the humans and machines such that they can do this in a very organic way, I think that's the missing link.

And I'm looking forward to seeing that come out as soon as possible. That kind of breaking it out of the chat prompt, right? Like getting it more integrated. The prompt is kind of a very one-dimensional version of that, but we need it way more. One of the things that has been very successful, so I come from a computer vision background.

background. And one of the early on lessons that we learned is that algorithms involving learning by example have always been very, very successful, right? And this reinforcement learning with human in the loop is just another example of this, right? Tell me whether you like A or B more. You don't have to tell me why, just stack rank them, right? In computer vision, it was like, here's an image with recognized objects.

Here's another image. Can you recognize any objects? I won't even tell you what these objects are, but they have to be similar to the objects in whatever dimension, color, shape, construct, family, like whatever. Can you recognize them? And we've been able to train machines to do this like 10, 15 years ago. So this learning by example is critical and machines have to start learning from humans by example, not at the training phase, but during actual inference of problem solving.

Interesting. Interesting. Yeah, this is great. Where can people follow you to learn more about what you guys are putting out or just to see what you have to say online? So there is an AI blog at Upwork. And whenever we publish things there, I usually cross-reference it with LinkedIn. Cool. Outside of that, we do a lot of foundational research. And when it gets published, you can see it on my scholar page.

Awesome. Well, Andrew, this has been a really enlightening conversation. I appreciate you making the time to share about Upwork and your work and point of view with our audience and love to have you back to check back in on things and see how things are going. It was great to talk to you, Luke. Thank you. Thank you very much. Have a good one. Thanks for listening to the Brave Technologist podcast. To never miss an episode, make sure you hit follow in your podcast app.

If you haven't already made the switch to the Brave browser, you can download it for free today at brave.com and start using Brave Search, which enables you to search the web privately. Brave also shields you from the ads, trackers, and other creepy stuff following you across the web.