PREDICTING ROAD USER TRAJECTORIES

One of the things that the autonomous vehicle’s designers (AI researchers, researchers in the fields of robotics, machine learning, agent modeling, human perception, etc.), need are rules and models that give them a way to predict what other road users will do. Since we are considered the experts in human behavior within the laboratory, it seems only natural for them to ask us to provide them with such models of social actions. We have been asked to provide, for instance, state transition diagrams to specify in detail the intentional states of road users and how they change their state as a result of changing circumstances. As an example, a particularly relevant thing for an AV to notice about pedestrians would be the change from “waiting” to “crossing the street,” or for a bicyclist the transition from “going straight” to “turning left.” Detailed observations of how these transitions occur could allow the AI researchers to construct models that would specify how the AV should act, given its perception of the environment.

While we acknowledge that for the engineers there is a necessity to break down the world into agents’ behavioral states and the transitions between states, the challenge for us as social scientists is that the reduction of road user behavior into transition diagrams is not only extremely difficult, but that even when such an attempt is made, something essential about the ‘social’ seems to be lost in the process. For instance, take the case of this pedestrian waiting to cross at an intersection in San Francisco.

PICTURE 2a & 2b. A pedestrian is waiting (2a) and follows a man who crosses before the light has turned to walk (2b).

Depicted in picture 2a & 2b is someone who is waiting among a group of people for the light to turn before crossing the street. The fact that he is among a group of people is relevant, since it makes it more likely that he might join the others in crossing a few seconds before the light turns to walk. One can try to break down a pedestrian’s behavior into its constituent parts, consisting of, for example, body posture, gaze direction, positioning on the road or sidewalk, but that break down seems to miss something essentially social and may not help you the next time you encounter a similar, but differently executed, social action.

Or take these two women in picture 3, who look like they are about to cross based on the way they approach the intersection (3a). Yet after they step into the street, they stop (3b), look around and point (3c), clearly engaged in an interaction to figure out where they should go. The analysis that they are “not crossing” would be highly relevant to an AV—and note that the car’s driver has easily figured this out and crosses (3d)—but depends on seeing that these women are not two individuals, but that they are together (the idea that walking down the sidewalk is dependent on an ability to see that certain people are together was the subject of early work in ethnomethodology [Lincoln Ryave & Schenkein, 1974]).

PICTURE 3a, 3b, 3c, 3d. Two women arrive at the intersection (3a). They step into the street to cross (3b) when the one on the left slows down and looks left. The woman on the right looks where her partner on the left is pointing (3c) as they halt at the edge of the road. The car that was waiting for them now crosses the intersection (3d).

Or, consider this picture 4 taken from the front of a moving car in southern California.

PICTURE 4. A woman steps out into the street behind a parked car.

The woman steps into the street with a large step behind a car. Considered simply as a moving object, as AV algorithms tend to do, she may represent a potential conflict for the car from which the picture was taken. Instead of a potential hazard, however, what we see is that she is the driver of the car parked on the side of the road and she is walking around it to get into it the left front door, where we know the driver sits in vehicles (this is in California). The woman standing on the sidewalk behind her will presumably ride shotgun adding to the Gestalt of a woman stepping out into the street to get into her vehicle.

To us as social scientists these examples demonstrate that social understanding imbues our understanding of street life, of people’s behavior in traffic, and of how we perceive the world in general. This social lens is paramount, and is both more than and qualitatively different from the raw sensory input (a ‘bit cloud’ from a Lidar sensor or a sequence of video images taken from a camera, for instance) from which the engineers must build up the AV’s interpretation of a scene. While it is certainly easy enough to discuss compelling examples with the engineers, it remains much more challenging to define the concrete implications of these social observations for their technical work.

MAKING PROGRESS

We have, nonetheless, had successes, not just in generating conversations across our disciplinary bounds, but also in tangible input to the development of the AV. An example of a successful result was our analysis of a road user practice we called “piggybacking.” This is a practice observed with some frequency at stop intersections. As stated, the traffic rules for four-way stops prescribe that cars may cross the intersection in the order in which they arrive and that when there is conflict the car from the right should go first. Piggybacking is a practice that systematically breaks this rule, but in a socially acceptable manner. Drivers who piggyback travel through the intersection by jumping ahead in the order, taking advantage of the priority established by a car ahead of them, often due to their place in the queue being blocked from going by another road user. The picture below helps to illustrate.

PICTURE 5. Piggybacking. The Blue car arrived earlier at the intersection than the yellow car, but the yellow car takes advantage of the fact that the red car is blocking the blue car from going.

Pedestrians too were seen to piggyback on the priority established by other pedestrians. A car may also piggyback on the right of way established by a slower moving bike. In other words, there are many ways that piggybacking occurs at four-way stops, resulting in subtle yet socially recognizable and socially acceptable ways in which the order in which road users cross may be altered.

Piggybacking was readily accepted by the technical team as the kind of social behavior they were eager to program into the perception and possible action of the AV. We believe this was because the behavior was recognizable and describable at the level of decision logic that was relatively easy for the AV engineers to implement; the AV had a concept of the queuing order, and thus was able to perceive and categorize the right objects for piggybacking (i.e., the order in which cars arrive at an intersection, and what constitutes a clear path through the intersection). Since it is a practice that breaks a driving rule in order to achieve better overall traffic flow, would it not constitute quintessential evidence that the AV was driving in a socially acceptable manner if it could piggyback? Moreover, while the public understands that AVs would drive conservatively and therefore safely, there is a great concern that their robotic driving style will impede traffic flow; piggybacking is a tangible example of how such concerns might be addressed.

MAKING FURTHER PROGRESS, THE INTENTION INDICATOR

Piggybacking wasn’t the only practice that stood out for us at stop intersections. We also noted that when there was apparent conflict about the order in which road users can cross, people often negotiate about who should go first, in particular between pedestrians and drivers. Pedestrians have the right of way in a crosswalk, but it is not clear whether they do when they wait at the curbside or only when they step into the road. Regardless, most pedestrians will not blithely step out into the crosswalk in front of an oncoming car. Pedestrians may seek eye contact with a car, or at least take a clear accounting of a car’s behavior before stepping out in front of it. We observed pedestrians that waved cars on, and drivers that waved on pedestrians. The pictures 6a, 6b, 6c, below illustrate one of these moments.

PICTURES 6a, 6b, 6c. Negotiation between a pedestrian and a driver at a four-way stop in town. In (a) the woman holds her partner back and waves the driver on, in response (b) the driver waves the pedestrians on. The pedestrians cross (c)

How could an AV interact and communicate with pedestrians, bicyclists and other drivers about the order of traffic? How could an AV express that it was letting a pedestrian go ahead, or, by contrast, that it was planning to go and that a pedestrian had better not step into the street? The AV should be able to express its intentions, but also take account of the specific situation and expectations of other road users in that setting, and adjust its behavior accordingly. We started to explore the hardware and software solutions that would allow the AV to communicate with other road users and developed a concept that we continue to adapt and refine.

The concept we developed involves a signaling system that would be added to the vehicle and could change modes according the interactional context when potential conflicts arise with other road users. With the system, AVs could engage in roadway negotiations based on the AV’s perception of what vehicles and other people on the road are doing and planning to do, and adapt its behavior in order to be a good ‘social partner.’ The concept involves a light strip viewable from the front and side of the car with programmable signals that communicate the cars intention to other road users in the vicinity.

One way we have begun to test the concept was to try it out in our facility on a remotely controlled toy car. The objective of these tests was to get feedback on the specific instantiations of the concepts – there were many ways to execute the details and we wanted to sort out how the different options would work. Which were the most easily understood, what unintended reactions did they provoke, and so on, focusing on whether the concept would have any meaningful bearing on interactions on the road (and ultimately traffic flow). We ran the experiment twice, in one case creating a simulation of an intersection on the road and asking people to take specific actions such as: cross the road as you normally would, linger before crossing, jaywalk, and so on. In the second test we did not direct people at all, but simply observed what people who moved around the lab experienced when they encountered the toy car.²

PICTURE 7. Employee interacting with the prototype remotely controlled car.

Although we had initially envisioned that the light strip would negotiate and interact with other road users in a way that would full-fill the function of hand waving, eye gaze, and the reading of ‘signals’ about other road users’ intent that we observed among pedestrians and drivers, at this point an actual interaction requires both a level of continuous perception and an ability to adapt the actions of the AV in real time that is far beyond the current system’s capabilities. Rather, our solution is for the light strip to display the AV’s intention. It has come to be called the Intention Indicator.

Despite these limitations the Intention Indicator concept was an immediate hit within the organization. Indeed it was taken up by the designers of the Nissan’s autonomous concept car, the IDS (Intelligent Driving System), and paired with other communication devices, including a technology that could detect pedestrians and bicyclists and could confirm that they had been ‘seen’ and an LED text display to reinforce messages to near-by road users. The IDS concept car was revealed at the Tokyo Motor Show in October 2015 (Picture 8).

Picture 8. The Nissan IDS concept car with the intention indicator represented by the blue light strip running along the sides and front of the car, as well as a display in the window.