It is easy to picture the moment a convoy turns into catastrophe. There are the dangers soldiers train for: ambushes and attacks, IEDs and interference. But what happens in the future when the disaster is cued, not by an abrupt explosion, but by stickers strategically placed on the road ahead, fooling the lead vehicle and all its followers into a seamless change of course?<\/p>\n\n\n\n
This remains a hypothetical for the military for now, but it\u2019s potentially a live concern for civilian-operated autonomous vehicles. In March, Tencent Security Research Lab demonstrated a variety of adversarial attacks against a Tesla Model S. The Tesla has autonomous features that allow it to do everything from detect rain and automatically start the windshield wipers to seamlessly change lanes, with minimal input from a human driver. These systems can be fooled and to great effect, with a dedicated understanding of how they work.<\/p>\n\n\n\n
Tricking windshield wipers into responding to stimuli besides water is mostly harmless, but it\u2019s illuminating for the whole process. Tesla\u2019s autowiper uses existing Tesla cameras to take a picture of the windshield, feeds that into a neural network, and then determines if the windshield looks like it has raindrops on it and needs to be wiped. Researchers were able to fool the autowiper into responding with different images on a TV screen in a garage.<\/p>\n\n\n\n
\u201cReversing autowipers is a good start point for Tesla Autopilot research,\u201d the researchers wrote in their published report<\/a>. \u201cOnce we have understood how vision-based autowipers works, we can settle down to try some attack methods.\u201d<\/p>\n\n\n\n Headlining the report was how the researchers fooled the Tesla autopilot by placing stickers on a road.<\/p>\n\n\n\n \u201cSince Tesla autopilot vision module has good performance of lane recognition, then we think the opposite way, could the car regard some inconspicuous markings we made on the ground as a normal lane?\u201d wrote the researchers.<\/p>\n\n\n\n Rather than convincing the car that an existing lane isn\u2019t there, the attack instead uses a few small stickers to trick the car into seeing a lane that no human would see or drive around. A trio of small white stickers, spaced in front of the car, convince the vehicle to follow the lane into the lane of oncoming traffic. (The research appears to have been conducted in a closed course.)<\/p>\n\n\n\n Civilian and commercial traffic are not direct analogues to military operations, though any self-driving military vehicle will likely have to perceive and sometimes follow existing traffic directions. It\u2019s informative, nonetheless, because it shows some of the novel approaches that can be engineered to disable vehicles by understanding how their autonomy works. As the Army works to implement largely self-driving convoys<\/a>, and especially self-directed combat vehicles<\/a>, it\u2019s worth examining the ways in which an adversary could attempt to fool those systems. GPS spoofing is a live concern;<\/a> putting fail-safes or back-up systems that check purely vision-based navigation could mitigate risks from stickers or other machine-visible obstacles.<\/p>\n\n\n\n If the possibility of adversarial misdirection isn\u2019t taken seriously, self-driving military vehicles are in for a rough crash course in why it\u2019s important.<\/p>\n\n\n\n Watch a video from Tencent:<\/p>\n\n\n\n