Last year, Waymo rolled out a truly driverless ride-hailing service in Arizona. This year, the US Air Force flew a fighter jet using AI for the first time. We could give you dispatches from uncrewed vessels on the high seas, but we don’t want to keep you here all day.
These stories tempt us. We’re prone to assume semi-intelligent systems are just about ready to make the leap to omni-capable autonomy. It can feel like a full-feature AI system is right around the corner, and that once it shows up on the scene, humans will become unnecessary.
But mind the fine text. Revisiting the earlier examples, both headlines have caveats galore:
- Waymo’s driverless vans have been training with in-car safety operators for years. Even now, they’re still monitored from afar by humans. And often, they’re monitored up close, as technicians follow in chase cars.
- The Air Force’s AI brain only performed specific in-flight tasks: “sensor employment and tactical navigation.” Had anything glitched, you can bet the human pilot would have stripped the AI pilot of its stripes mid-flight.
By the day, flying, roving, and package-sorting robots are getting smarter. We’re granting more permissions to machines like drones and autonomous vehicles and letting them out of their sandboxes. On some specific benchmarks—such as vision—algorithms already have us beat. But robots still can’t do the generalized tasks that we can. One of the most elusive frontiers of automation is building machines that can intelligently and autonomously move around unstructured, real-world environments. Think about sidewalk delivery bots, an increasingly common fixture of US streets, which still have trainers standing by (from up close or afar).
As robots across the spectrum of automation graduate, even the class stars need human minders to debug code, audit errors as they occur, and ensure they’re staying on task in the physical world. And by nearly any insider’s account, humans aren’t leaving the picture anytime soon.
An anthropological look at human-machine relations
The loopty loop paradigm* is an important way to consider the degree of automation in software, algorithms, and physical robots. Automation is more like a gradient than a ladder, but for simplicity’s sake, you can think about the loopty loop in buckets:
- Human out of the loop = A robotic or artificially intelligent system makes decisions without the need for humans.
- Human in the loop = A person is required for this type of system; They have full control over starting and stopping it.
- Human on the loop = Popularized by the Pentagon, this term refers to a system where humans are further removed from automated decision-making. They can intervene, but it is not absolutely necessary.
- *This classification scheme is not actually known as the loopty loop paradigm. Unless, that is, readers join our campaign to inject some fun into human-machine interaction studies.
Case study: air vs. land
In the eyes—er, sensors—of an automated system, the roads have more obstacles than the skies. Compared with AVs, drones have a separate risk profile and set of physics challenges, but it’s less congested up there. (Try as you might, you will not find any angry Massachusetts motorists 400 ft. above ground level.) Even taking air lanes and flight paths into account, drones aren’t constrained to forward and back. They can go up, down, and side to side. They’re not confined to the places where we’ve poured asphalt.
“We now know that the drone has the capability to fly around by itself and land,” Colin Guinn, the former CEO of DJI North America, told us, along with autonomously avoiding obstacles, snapping photos, and stitching them together into a 3D model/digital twin. Guinn estimates that 90% of commercial drone flights are little more than remote pilots “loading a flight plan, pressing a button on an iPhone, and watching a drone do its thing.”
With few exceptions, US regulators require pilots today to maintain a visual line-of-sight with their drone. A company needs one pilot for every drone they’re actively flying. It’s a 1-to-1 relationship.
- The bad news for these pilots is that autonomous flight systems can already do nearly all of their job.
- “The pilot is the biggest expense,” Guinn said. “You have to pay them $150 to drive to the site, then mark that up by 4x and sell the aerially produced digital model to a client.”
As the FAA inches toward relaxing line-of-sight requirements for commercial drone operations, “These autonomous robots can act as autonomous robots,” Guinn said. That could slash demand for drone pilots, but it won’t make their skillset obsolete.
“At no point in the next decade will drones be fully out-of-the-loop systems,” he added.
Back to the landlubbers
Overly optimistic AV deployment timetables have famously come crashing back down to Earth in recent years. Some industry executives say that the human legwork required to teach a car to do what we can already do—drive—is harder than rocket science. Still, the self-driving industry is far from dead in the garage. For now, let’s set aside cars and focus on a more specific form of autonomous vehicle…
…like forklifts. Phantom Auto is a Bay Area-based business that sells software for remote monitoring, assistance, and operation of semi-autonomous vehicles. The assistance category involves operators giving commands to a robot and/or drawing a path via navigation waypoints.
- “Operation” refers to remotely driving the vehicle. This Phantom business line is mostly forklift drivers.
Elliot Katz, Phantom Auto’s cofounder and chief business officer, cites three demand catalysts for teleoperation services.
- Health/safety: For an employer, Covid meant reducing colocated employees as much as possible. A non-pandemic incentive for removing forklift operators from warehouses = It’s a dangerous job. “With remote operation, customers move operators physically out of the warehouse and thus out of harm’s way,” Katz said.
- Labor accessibility: There’s a shortage of available workers. Remote operation is a workaround that helps eliminate geographical constraints and widens the available talent pool.
- Productivity: Remote operation removes friction, because “you can essentially have…digital drivers teleport to a new warehouse,” Katz said.
If customers want an autonomous forklift, they’d need it to operate at the same level as a human. “If a forklift gets stuck, that’s not deployable,” Katz said.
Takeaway? “The fact of the matter is that you can never eliminate the human from the loop,” Katz added.
The Tony Stark option
Formant, an SF startup that sells a robotic fleet management platform-as-a-service, recently let us take a Boston Dynamics Spot robot for a spin from ~1,700 miles away. While teleoperation is cool, “it’s actually the smallest part of our product,” CEO Jeff Linnell told us. Formant’s monitoring service—“like Tony Stark view for robots”—is where customers spend 90% of their time, Linnell said.
“Literally, joysticking is like the sensational example of yes, you can drive a robot from Texas,” Linnell says, but it’s not sustainable or scalable, he said. “The more interesting question: ‘Can I keep 100 robots working on production sites?’”
“Robotics is supposed to have happened for the last 20 years,” Linell said, but it hasn’t. “It’s because nobody’s worked on the dialogue between the robot and the people that need to manage it.”
Robots are good with precision, repeatability, strength, and endurance. They can lift more than you, and no amount of gym visits will ever change that.
But humans are still needed for what they’re good at, “problem-solving, intellect, understanding, and broadcasting intention,” Linnell said. “If a robot gets 95% accuracy on a task like unloading a truck, if you involve a human in it, to take the ambiguity out of it, we’ll have 100%.”
Add it all up, what do you get?
When it comes to the gradient of automation, the Society for Automotive Engineers’ (SAE) six levels of autonomous driving are instructive. You’re fully in the loop for a Level 0, “no automation,” vehicle. You’re still in the loop for “conditional automation,” or Level 3. If we’re being sticklers for terms, even at fully autonomous Level 5, a human is still briefly needed in the loop. After all, unless it’s a mind reader, the car needs to know where you’re going.
From software to industrial robots, we do have historical evidence of machines eliminating entire job categories. While first- or second-wave AI systems may need progressively fewer overseers, they’re still not running on autopilot. At the n-wave—i.e., ubiquitous robots, AVs, and drones operating in unconstricted, non-structured—millions of jobs would be displaced. Cockpits and truck cabins would go the way of the horse and carriage.
Companies are spending tens of billions in search of the n-wave; Militaries, too, are investing heavily on programs to disintermediate human crews from warships and fighter jets.
But there are many, many steps to get to mainstream deployments of those advanced AI systems. And each step involves ample levels of human agency and decision-making. Beyond software engineers or QA technicians, we can see an early version of a future in-demand specialist job today: the remote operator.
In centralized digital driving centers or drone dispatch hubs, the operator-machine ratio will evolve from 1-for-1 to 1-to-many. But not 0-to-many.