By Charlie Keohan and Nicholas Kralev
August 7, 2022
If you want to work for the U.S. Department of State as a Foreign Service officer, your application must now be screened by an artificial intelligence model before you can advance to the next stage of the selection process. The software took over this critical function from a written test, which served as a “pass/fail gateway” for decades, until earlier this summer, department officials said. You still have to take the test, but your score will now be combined with a score the A.I. will give your application, based on your qualifications and six essays. If you make it past that first cut, your file will be reviewed by humans, before you can advance further in the process.
“The deep textual analysis software is based on thousands of scored essays reviewed by our assessors, who base their scoring on 13 dimensions,” said Jefferson Smith, the State Department’s director of talent acquisition. The dimensions include composure, cultural adaptability, information integration and analysis, judgment and objectivity.
The A.I. screening is part of the second phase of the lengthy Foreign Service selection process, which comes after the Foreign Service Officer Test (FSOT) and is known as the Qualifications Evaluations Panel (QEP). The software evaluates candidates’ educational and professional qualifications, as well as the essays, which the department calls “personal narratives.” They “describe the knowledge, skills and abilities the candidate would bring,” based on “core precepts that are predictors of success in the Foreign Service: leadership, interpersonal, communication, management, intellectual skills and substantive knowledge.”
Applicants are ranked based on the combined score from the “computer-QEP” and the FSOT, and the top tier in each career track — political, economic, consular, management and public diplomacy — then advances to an “assessor-QEP.” The assessors, who are members of the Board of Examiners, manually score those files, and the top candidates are invited to the Foreign Service Oral Assessment (FSOA), officials said. That number may vary by career track and selection cycle, based on the department’s budget and hiring needs.
“This is the most significant change to the Foreign Service selection process since 1930,” the department said when it announced the reform in April. “We anticipate that this change will result in identifying a more qualified pool of candidates.” It’s aimed at increasing diversity in the service, Brian McKeon, deputy secretary of state for management and resources, told the Senate Committee on Foreign Relations in May.
Thousands of Americans apply to become Foreign Service officers every year, but the State Department accepts only a few hundred. The FSOT is offered three times a year, usually in February, June and October.
So how does the software work? It’s not entirely clear, which concerns the American Foreign Service Association (AFSA), the diplomats’ union. “How are these algorithms written?” asked Eric Rubin, AFSA president and a former ambassador to Bulgaria. “What do they include and what do they not include?” The department has “not been sufficiently transparent on how the algorithm works and how it will improve itself,” he said.
Smith said that “the deep textual analysis software is proprietary,” because it was written by a private company, not U.S. government employees. He also took issue with the word “algorithm,” noting that the software isn’t programmed to look for specific keywords or phrases. Rubin said that the software being “proprietary to the contractor is not really OK.” “This is federal hiring. This is not Amazon or McDonalds,” he added. “This is public money, and this is a public contract, and this is public hiring.”
Smith noted that the software was first introduced in 2015, but it was used only after the FSOT made the first cut, so it was less significant. However, Rubin said the State Department kept it secret until it announced the new changes in April. “Neither AFSA nor members of the Foreign Service knew about the use of A.I. until this current set of reforms,” he said. “We were never informed about this — until now.”
Having failed to give AFSA advance notice about the changes in the selection process — or to consult it — the department later apologized. “We have been briefed now,” Rubin said. Once in the service, most officers can stay in for at least 20 years, so “we think that how people are chosen is very important, and the public and AFSA have a right to know and understand that.”
“We understand that there needs to be a gatekeeper function, and there are simply too many applicants” for every essay to be reviewed by a Board of Examiners assessor, Rubin added. “We know that part of the algorithm is based on previous successful applicants, which makes sense. But we would like to know more, and to see numbers and outcomes on how it’s going to be working.”
Smith said that the “assessors regularly test the program’s reliability and validity to ensure its accuracy, further improving its effectiveness by adding more scoring data with each cohort.”
In general, Rubin said, AFSA appreciates the State Department’s efforts to modernize the Foreign Service officer selection process. “Our entire service is based on legislation that is 42 years old,” he said in reference to the Foreign Service Act of 1980.
“Just about every aspect of the current Foreign Service needs reform,” he said. “So, in principle, we are supportive. We are cautiously saying, ‘Okay, let’s see how this goes.’ But we expect it to be reviewed soon and regularly.”
