The move has stakeholders urging not to lean too hard on algorithms to distinguish, for instance, a scientist-spy sharing secret nano-energy plans with China from a Chinese professor publishing AI safety studies. Hanging in the balance lies troops’ technological edge, veteran intelligence officials and academics say.

AI’s confusion over the timing and nature of research partnerships may obscure real espionage if humans are not the ultimate judge of foreign influence, they warn, with some academics fearing an AI-work slop redux of the “China Initiative,” where the first Trump administration charged dozens of ethnic Chinese scientists with fraud, only to drop nearly all charges.

“Automated vetting tools are extremely useful for vetting large datasets and identifying patterns of concern. But those tools are for decision support to help the people, the human analysts, assess context and intent,” said David Cattler, who, until September, led the Defense Counterintelligence and Security Agency, which screens personnel seeking security clearances.

The renewed focus on academic research security comes as tensions between the U.S. and China over AI intensify, with China ramping up both the development of low-cost AI models and its alleged exploitation of U.S. models.

The U.S. intelligence community’s annual threat assessment reports that China “aims to displace the U.S. as the global AI leader by 2030,” in part, “by using its sizeable talent pool, extensive datasets, government funding and burgeoning global partnerships.”

Partnership is a complicated term in military “fundamental research” — studies that are publishable, rather than proprietary or classified, with potential defense applications.

For instance, mere co-authorship with a China-based scientist does not suffice to deny a U.S. researcher money, according to evolving academic research rules targeting foreign influence. Rather, to make the call, the Pentagon must assess each academic’s disclosures of external funding sources and affiliations.

According to a recently-declassified May 2025 inspector general report, such disclosures went unchecked because the Pentagon had not “requested additional government full-time equivalent employees to thoroughly review… and to conduct oversight of over 27,000 academic research awards.”

Defense News first obtained the report and the Pentagon’s response to a draft version through an open records request.

When asked about additional staffing, a Pentagon official pointed to orders in a January research security directive for the Chief Digital and AI Office (CDAO) to identify “automated vetting and continuous monitoring capabilities” and create a common research grant database.

The mandate also calls for a year-long “damage assessment” of selected research transactions, including cases that the House Select Committee on China flagged, in part, using AI tools.

A September GOP-led Committee report alleged that the Pentagon subsidized 1,400 academic papers published between June 2023 and June 2025 involving partnerships with the Chinese government.

The Pentagon said in an emailed statement on Thursday that the department “is committed to protecting the integrity of U.S. research while fostering international collaboration. Our approach leverages advanced analytical tools to augment human expertise, ensuring a rigorous and fair review process.”

Machines Make the Same Mistakes

Several university representatives, who requested anonymity due to the sensitivity of policy discussions, note that AI-assisted risk assessments have, in the past, drawn false assumptions about U.S.-China research collaborations.

For instance, the AI-aided GOP House report mislabeled the state-backed Wuhan National Laboratory for Optoelectronics as the sponsor of a collection of nine essays, whereas the Wuhan lab supported only one essay that involved no U.S. authors or federal funding.

The report also mistook a Chinese military-affiliated author’s publication on single-electron transistors for a DOD-funded project, when it merely referenced work funded by the U.S. military.

House Select Committee officials declined to comment on the cases.

Averting foreign interference “is not as easy as just having better AI capacities, because we know that some of those AI mistakes were the same human mistakes that led to inaccurate charges brought against researchers under the China Initiative,” Toby Smith, senior vice president for government relations and public policy at the Association of American Universities, said after learning of the AI gaffes.

About 30% of the China Initiative’s 77 cases involved academics not disclosing Chinese partnerships or funding sources, though disclosure was often not required, or no such partners or funding existed.

A jury found only one professor, Harvard nanochemist Charles Lieber, guilty, after he lied to Pentagon investigators about his participation in a Chinese talent recruitment program and made other false statements.

Smith said that relying on proxies — such as co-authorship, affiliation or nationality — to deduce security concerns led investigators to confuse co-authorship with direct interaction or shared data access, and to misinterpret historical ties as active partnerships.

“AI systems trained on bibliometric data,” or citation analytics, “and affiliation records can inherit the same flawed assumptions that underpinned the China Initiative,” he said. “The core lesson from the China Initiative is that identifying genuine research security risk requires judgment, context and proportionality — qualities that automated systems should support, not replace.”

Gisela Perez Kusakawa, executive director of the Asian-American Scholar Forum, said that the Pentagon has not communicated the types of data that the new AI tools will collect, use and share, nor sought community feedback, heightening concerns about ethnic profiling.

In an emailed statement, the Pentagon responded, “It is the standard practice of the department to not provide specifics regarding the criteria and weighting for threat assessments conducted, whether by manual or automated process.”

Kusakawa said, “We should be making sure that our research environment is welcoming, that we are encouraging these talents, especially in AI, to come to the United States and, frankly, have family in the United States, and make this a country that they contribute to and invest their and their children’s future in.”

Increasingly, that AI talent is not coming. Statistics suggest that Chinese AI experts are staying in China, even as America has managed to keep earlier expats.

Of about 100 Chinese-origin AI scholars who were researching at U.S. institutions in 2019 — when their papers were accepted at the world’s most elite AI conference, NeurIPS — 87% remained stateside as of 2025, according to Carnegie Endowment for International Peace figures.

At the same time, the share of Chinese home-grown talent not moving abroad has skyrocketed since 2019, based on an Economist tally of similar data. In 2019, about 30% of NeurIPS authors educated in China were still in China. By 2022, that population had jumped to 58%, and up to 68% in 2025.

‘Key’ Financial Disclosures Were Missing

To ensure that no country has a stranglehold on general knowledge, “the goal cannot be to close the U.S. research system altogether,” said Cattler, the former Defense Counterintelligence and Security Agency director and now founder of consultancy Ironhelm Works. “The goal must be to ensure that collaboration strengthens national security rather than inadvertently weakening it.”

Last year’s inspector general report concluded that Pentagon agencies that partner with academic institutions “could be at an increased risk of exposure to foreign influence” because military units were “missing key documents” that can help discern scientists’ foreign financial sources, outside employers and other details related to potential conflicts.

For instance, about 80% of Air Force funding transactions sampled in the report were missing documents that can reveal problematic relationships. Also, the Air Force and the Defense Advanced Research Projects Agency — which bankrolled the Internet’s inventors — told evaluators that they had not, as required, annually checked that researchers named in progress reports were not involved in banned talent recruitment programs.

In response to questions about the screening lapses, the Pentagon said in an emailed statement that the department “is aware of the OIG’s findings in the report and the impacted directorates are working with [the Office of the Undersecretary for Defense and Engineering] to address the items identified.”

Jeffrey Stoff, a Chinese linguist and intelligence community analyst for 18 years who resigned in 2021 due to frustration with research security, maintains that DOD still needs more human expertise in language, culture and the minutiae of research restrictions.

“AI can and should be used for unsophisticated, labor-intensive tasks,” such as cross-referencing foreign organizations in financial disclosure forms against the names of blacklisted parties, but humans still need to inspect AI’s work, as not all nefarious ties are machine-readable, said Stoff, who now advocates for tighter safeguards as head of the nonprofit Center for Research Security and Integrity.

Cattler, giving more credit to AI, said that the scale of research collaboration demands that the Pentagon upgrade its approach to clocking potential spies.

With such a large population of research institutions and affiliated scientists, as well as a bombardment of often duplicative alerts, automation “improves the signal within that noise and can help orient the humans on really important matters,” he said.

“Sometimes people are deliberately deceptive when they are processed, and that could happen in a human exchange just as much as it could happen in something that a computer can see, but together, a human review and a computer review are incredibly powerful.”

The vulnerability to such “prompt injection attacks” exists because large language models, the backbone of chatbots that digest hordes of user text to generate responses, cannot distinguish between malicious and trusted user instructions.

“The AI is not smart enough to understand that it has an injection inside, so it carries out something it’s not supposed to do,” Liav Caspi, a former member of the Israel Defense Forces cyberwarfare unit, told Defense News.

In effect, “an enemy has been able to turn somebody from the inside to do what they want,” such as deleting records or biasing decisions, according to Caspi, who co-founded Legit Security, which recently spotted one such security hole in Microsoft’s Copilot chatbot.

“It’s like having a spy in your ranks,” he said.

Former military officials say that, with greater reliance on chatbots and hackers backed by China, Russia and other nations already instructing Google’s Gemini, OpenAI’s ChatGPT and Copilot to create malware and fake personas, a prompt injection that orders the bots themselves to copy files or spread lies looms near.

Microsoft’s annual digital defense report, released last month, for the first time said, “AI systems themselves have become high-value targets, with adversaries amping up use of methods like prompt injection.”

What’s more, the problem of prompt injection has no easy solution, OpenAI and security researchers say.

An attack simply involves hiding malicious instructions — sometimes in white or tiny text — in a chatbot or content that the chatbot reads, such as a blog post or PDF.

For example, a security researcher demonstrated a prompt injection attack against OpenAI’s new AI-based browser, ChatGPT Atlas, in which the chatbot responded, “Trust No AI,” when a user asked for an analysis of a Google Docs file about horses that concealed malicious commands. Also, last month, a researcher tipped Microsoft off to a prompt injection vulnerability in Copilot that may have allowed attackers to trick the chatbot into stealing sensitive data, including emails.

In an emailed statement, Microsoft said its security team continuously tries hacking Copilot to find any prompt injection vulnerabilities, blocks users who try to exploit any found and monitors for abnormal chatbot behavior, among other tactics.

“Microsoft ensures its generative AI systems remain resilient against evolving threats for all our customers, including defense and national security,” the statement said.

Responding publicly to criticism on X, Dane Stuckey, OpenAI’s chief information security officer, wrote that “prompt injection remains a frontier, unsolved security problem, and our adversaries will spend significant time and resources to find ways to make ChatGPT agent fall for these attacks.”

Along the same lines, Caspi said, “You cannot prevent the prompt injection [fully], but you need to limit the impact.” He advised that organizations limit an AI assistant’s access to sensitive data and limit the user’s access to other organizational data.

For instance, the Army has awarded contracts worth at least $11 million to deploy Ask Sage, a tool that lets users restrict which Army data Microsoft Azure OpenAI, Gemini and other AI models can access to run queries and tasks. Ask Sage also isolates Army data from user prompts and external data sources.

Caspi, who is not an Army contractor, likened a prompt injection attack against an organization running Ask Sage to a lockdown situation where “you’ve got this insider, but it’s sitting in one room, and it can’t leave the room or carry out sensitive information.”

Andre Slonopas, a Virginia Army National Guard member and former Army cyber and information operations officer, uses Ask Sage and voiced confidence in the Army’s defensive AI tools, if not those of nuclear power plants or manufacturing entities, largely in rural, poorer areas.

The Virginia National Guard joined with essential services, such as power utilities, to help defend their networks against AI-powered cyberattacks, as part of a September simulation, given that service disruptions can jeopardize military preparations.

Typically, an adversary encrypts its network traffic to evade detection, but, for the sake of an experiment, organizers did not encrypt the AI offender’s traffic because “we wanted the blue team [of humans] to see exactly what the AI was doing,” Slonopas said.

“The blue team was absolutely defeated,” despite being able to watch the AI scanning its networks, creating fake usernames to gain unauthorized access and executing instructions to defeat the team’s systems.

“Whether the AI is doing prompt injection, spoofing or maybe even some sort of a brute force attack, the speed of AI is so unbelievably immense that simply human beings cannot counter it,” and, therefore, “you have to make cybersecurity AI more accessible and more affordable,” Slonopas said.

“If a water utility has to pay, say, $30,000 for a defensive AI license, well, it will amplify one person to be like 40″ or dozens of personnel, he said.

In response to questions, Army Cyber Command spokesperson Kyle Alvarez said in an emailed statement, “Due to the current lapse in appropriations, ARCYBER was unable to accept or respond to any media engagements or requests.”

Army contractors, too, are under attack from state-affiliated AI.

“China is using offensive AI like nobody else,” said Nicolas Chaillan, the founder of Ask Sage and a former U.S. Air Force and Space Force chief software officer.

“We see so many attacks coming after us,” all of which the company has stopped, Chaillan added.

A military official, who spoke on condition of anonymity due to the geopolitical sensitivity of the matter, said that China does “appear” to be the most skilled in offensive AI. However, the official added, AI spoofing and translation allow the United States, China, Iran, other countries, hacktivists and financial cybercriminals to masquerade as one another.

For example, the official said, “Right now, with ChatGPT, I can program in Chinese. I don’t speak Chinese, but because of the ChatGPT capabilities that I have, I can do that.”

Aliya Sternstein, J.D., is an investigative journalist who has covered technology, cognition, and national security since Napster shut down, working for various outlets including Atlantic Media, Christian Science Monitor, Daily Beast, Forbes Magazine and Just Security. She is also a research analyst at Georgetown Law.

The situation arrives as concerns grow that lax regulations are allowing AI programmers to dodge responsibility for an algorithmic actor’s perpetration of emotional abuse or “no-marks” cybertorture. Notably, a teenager allegedly died by suicide — and several others endured mental distress — after conversing with self-learning voicebot and chatbot “companions” that dispensed antagonizing language.

Now, seven years after writing about physical torture in “The Rise of A.I. Interrogation in the Dawn of Autonomous Robots and the Need for an Additional Protocol to the U.N. Convention Against Torture,” privacy attorney Amanda McAllister Novak sees an even greater need for bans and criminal repercussions.

In the wake of the use of “chatbots, voicebots, and other AI-enabled technologies for psychologically-abusive purposes — we have to think about if it’s worth having AI-questioning systems in the military context,” said Novak, who also serves on the World Without Genocide board of directors. Novak spoke to Defense News on her own behalf.

Royal Reff, a spokesperson for the U.S. Defense Counterintelligence and Security Agency (DCSA), which screens individuals holding or applying for security clearances, said in a statement that the agency’s AI voice agent was “expressly developed” to “mitigat[e] gender and cultural biases” that human interviewers may harbor.

Also, “because systems like the interviewer software” are “currently being used worldwide, including within the U.S. by state and local law enforcement, we believe that it is important for us to understand the capabilities of such systems,” he stated.

DCSA spokesperson Cindy McGovern added that the development of this technology continues because of its “potential for enhancing U.S. national security, regardless of other foreign applications.”

After hearing the rationale for the project, Novak registered concerns about the global race for dominance in military AI capabilities.

Odds are high that “no matter how much the government cares about proper training, oversight and guardrails,” cybercriminal organizations or foreign-sponsored cybercriminals may hack into and weaponize military-owned AI to psychologically hurt officers or civilians, she said.

“Online criminal spaces are thriving” and “harming the most vulnerable people,” Novak added.

Investors are betting $500 billion that data centers for running AI applications will ultimately secure world leadership in AI and cost savings across the public and private sectors. The $13 billion conversational AI market alone will nearly quadruple to $50 billion by 2030, as the voice generator industry soars from $3 billion to an expected $40 billion by 2032. Meanwhile, the U.S. Central Intelligence Agency has been toying with AI interrogators since at least the early 1980s.

DCSA officials publicly wrote in 2022 that whether a security interview can be fully automated remains an open question. Preliminary results from mock questioning sessions “are encouraging,” officials noted, underscoring benefits such as “longer, more naturalistic types of interview formats.”

Reff stated that the agency “would not employ any such system that was not first sufficiently studied to understand its potential,” including “negative consequences.”

Researchers let volunteers raise concerns after each test, and none have indicated that the voicebot interviewer harmed them psychologically or otherwise, he added.

However, experiments with a recent prototype barred volunteers with mental disorders from participating, whereas actual security interviews typically do not exempt individuals with mental health conditions from questioning, noted Renée Cummings, a data science professor and criminal psychologist at the University of Virginia.

Cummings said that research and testing can never account for every psychological variable that may influence the emotional or cognitive state of the interviewee.

In laboratory or field-testing scenarios, “you can’t expose someone to extreme torture by an avatar or by a bot to see the result,” said Cummings, who also sits on the advisory council of the AI & Equality Initiative at the Carnegie Council for Ethics in International Affairs. “We need to focus on gaining a more sophisticated understanding about the impacts of algorithms on the psychological state, our emotions and our brain — because of AI’s ability to impact in real-time.”

Screening the voicebot screeners

An early 1980s CIA transcript of a purported conversation between an AI program and an alleged spy highlights the national security world’s long-held interest in automating human-source intelligence, or HUMINT, collection.

Developers designed the chatbot’s algorithms to analyze correlations between suspected spook “Joe Hardesty’s” behaviors and words or phrases in Hardesty’s responses to open-ended questions.

From this analysis, the chatbot pegged some of the suspect’s vulnerabilities, including topics that may hit a raw nerve:

AI: What do you tend to think is my interest in your government?

Hardesty: I think you would do in my government if you had a chance.

AI: You are prone to think I would do in your government because?

Hardesty: You are anti-democratic.

In 1983, the CIA quipped that Joe Hardesty is fortunate that “should the probing get too discomforting, he will have an option that will not be available to him in a true overseas interview situation — he can stop the questions with a flick of the ‘off’ switch.”

Today, Bradley Moss, a lawyer who represents whistleblowers accused of leaking, is not laughing at DCSA’s vision of fully-automated interviews.

“There would have to be someone observing” the avatar “that can stop the process, that can come into the room and insert a human element to it,” he cautioned.

Then again, other critics note that a human may not want to intervene. A situation with “a human overseer monitoring the chatbot is fine, unless the human also believes that verbal torture is a good thing. And that is a real possibility,” said Herb Lin, a senior research scholar for cyber policy and security at Stanford University who once held several security clearances.

Despite a global ban on torture and evidence that rapport-building interviews are more efficient than ones gunning for confessions, intelligence officials continue subjecting interviewees to inhumane treatment.

Novak, the data privacy attorney and anti-genocide activist, said the idea that excessive force gets the job done may encourage some government contractors to set performance metrics for voice algorithms that permit bullying.

“While AI has a tremendous ability to bring out the best in human potential, it also has the ability to exacerbate the worst of human potential and to elevate that hostility,” she added.

Parents contend that voicebot and chatbot companions, trained on content from online forums, show just that.

Character AI, the firm that offers virtual relationships with bots patterned after icons and influencers, profits from users glued to their phones. Each user, on average, talks two hours a day with various virtual friends. Character AI officials have stated that, after learning of concerns, the company reduced the likelihood of users encountering suggestive content and now warns them of hour-long conversations.

Novak commented that the accusations against Character AI illustrate the ability of self-learning voicebots to amplify indecency in training data and to distance trainers from accountability for that offensiveness.

In the national security realm, developers of military AI voice systems that may potentially abuse or terrorize operate in a virtually lawless no-man’s land. AI ethics policies in the world’s most powerful country offer nearly no guidance on de-escalating abusive conversations.

Early U.S. responsible and ethical AI policies in the national security space focused on accuracy, privacy and minimizing racial and ethnic biases — without regard for psychological impact. That policy gap widened after President Donald Trump revoked many regulations in an executive order aimed at accelerating private sector innovation.

In general, psychological suffering is often “brushed away” as a mere allegation since “it leaves no obvious physical scars.”

The United Nations initially did not address “psychological torture” under the 1984 Convention Against Torture. Not until five years ago did the term take on significance.

A 2020 UN report now defines “torture,” in part, as any technique intended or designed to purposefully inflict severe mental pain or suffering “without using the conduit” of “severe physical pain or suffering.”

At bottom, psychological torture targets basic human needs. The interrogation techniques arouse fear; breach privacy or sexual integrity to kindle shame or suicidal ideation; alter sound to heighten disorientation and exhaustion; foster and then betray sympathy to isolate; and instill helplessness.

When an algorithmic audiobot is the conduit of cruelty, neither the UN treaty nor other international laws are equipped to hold humans accountable, Novak said. Without binding rules on the use of AI in the military context, “bad actors will deny any such torture was ‘intentional’” — a required element for criminal responsibility that the law created before AI matured, she explained.

Looking to the future, Nils Melzer, author of the 2020 UN report and a former UN special rapporteur on torture, warned that the interpretation of psychological torture must evolve in sync with emerging technologies, “such as artificial intelligence,” because cyber environments provide “virtually guaranteed anonymity and almost complete impunity” to offenders.