Considering its current capabilities and its potential for the future, ChatGPT is already impacting many professions and disciplines, including information technologies, data analysis, healthcare, financial services, customer service, education, and marketing. Released at the end of 2022 by OpenAI, this chatbot powered by the GPT-3.5 large language model (LLM) is designed to use generative AI and natural language processing (NLP) to produce text that, in some cases, seems to be indistinguishable from what a person might produce.
It has been trained on a massive dataset of information from the internet, which enables it to address a wide range of topics and provide coherent and relevant responses to a diverse set of questions. The model is designed to answer questions, engage in conversation, and perform various natural language processing tasks, making it a useful tool for various applications, such as customer service, content creation, and language translation. Of course, one of IKM’s primary interests is ChatGPT’s potential impact on performance in high-tech roles.
In this paper we explore the implications of ChatGPT for high-tech job roles from two perspectives, it’s:
- use as a job aid and potential to significantly disrupt several high-tech occupations, and
- impact on reliability and future content of employment tests for high-tech roles
We maintain, while ChatGPT and related technologies will not eliminate many high-tech roles as may be feared, they will continue to change the way key tasks in high-tech roles are performed. This will continue to have an impact on how candidates and employees are assessed for hiring and professional development.
Job Aid, Disruptor, or Both?
Despite the warnings not too rush in, the impact has been noticeable. Fortune and resumebuilder.com report on a survey of 1,000 employees, from which they find that nearly half of business leaders have already implemented ChatGPT in some aspect of the business, including code writing, copywriting, content creation, and customer support. From this same survey, about 50% of the business leaders who have used ChatGHPT, suggest that it has already replaced employees in their company.
How much disruption will ChatGPT cause in high-tech roles such as programmer, software developer, data scientist, computer and information system manager, etc.? There has been a lot of buzz about the potential of ChatGPT to write code in various programming languages such as Python, Java, C#, C++, Javascript and Go. For example, it can take a natural language description of functionality to be implemented and generate sample code snippets based on that description. It can also be used for such tasks as quickly generating frameworks, and outlining builds of applications, giving input into questions such as how data should be structured and what user interface features are needed. This technology promises to have the biggest impact on role performance in the following areas:
- Code Generation: ChatGPT can be used to generate computer code based on natural language input. This could assist with programming tasks by automating the process of writing code, freeing up more time for developers to focus on other tasks. Additionally, templates based on common coding patterns and best practices can provide a starting point for larger projects.
- Testing and Debugging: This technology will assist with testing and debugging by analyzing code and providing recommendations for improvements. This could help catch coding errors early in the development process, identify areas where the code could be optimized, and generally reduce the amount of time developers spend on manual testing and debugging.
- Documentation: ChatGPT will help will documenting code and software by generating natural language descriptions of code functionality and usage. This will improve the readability and usability of code, making it easier for other developers to understand and use.
- Cybersecurity: This technology is capable of identifying potential threats or vulnerabilities improving the effectiveness of cybersecurity measures and reducing the cyber-attack risk.
- Natural Language Processing: ChatGPT will improve NLP for IT applications to improve voice recognition, chatbots, and other applications that rely on natural language input.
In addition to benefits for programmers, ChatGPT holds the promise of improving efficiency and effectiveness in many job roles across a wide range of industries. Industries that IKM is watching closely include:
Healthcare: In healthcare, ChatGPT can enhance patient outcomes by improving:
- Diagnostics and Treatment Prescription: By inputting patient symptoms and medical history, ChatGPT could generate a list of potential diagnoses and recommended treatments. This could be useful in situations where medical professionals need to quickly identify potential diagnoses and treatments.
- Patient Communication: This technology can be used answers to common questions and concerns. It also shows promise in facilitating patient intake and scheduling, as well as providing health education materials.
- Electronic Medical Record (EMR) Management: It can automatically transcribe notes from patient visits and update patient records with relevant information. This could reduce the time and effort required to document patient information, freeing up more time for direct patient care.
- Medical Research: It can analyze large amounts of medical data more quickly and accurately than humans, helping researchers identify patterns and relationships that might not be immediately apparent, leading to discoveries and insights.
Banking & Financial Services: ChatGPT promises to be an effective job aid in such areas as both client management and analytics, including:
- Personalized Financial Advice: This technology will provide personalized financial advice to customers based on their financial data and goals. This can include recommendations for investment strategies, savings plans, and debt management.
- Compliance and Risk Management: This technology can provide information on regulatory requirements and potential risks associated with different financial products and services.
- Financial Analysis and Reporting: This technology can also be used to assist with financial analysis and reporting. It can generate financial reports, perform data analysis, and assist with forecasting and budgeting.
With respect to the degree of expected disruption, the effects will be minor enhancements to efficiency in some areas, for other it may change the way some key tasks are performed, and certain roles may experience significant changes in the way the role is performed. Customer Service roles might seem especially ripe for this type of disruption. The emotional labor associated with this occupation is well documented. The Wall Street Journal recently reported on a 2022 survey, showing that as much as 65% of call center agents anticipated leaving their jobs within the following two years. This is consistent with other data from the call center environment suggesting a high level of disengagement and burn out in this occupation. By scanning and analyzing thousands of conversations , AI has been able to identify patterns in the words and sentiments between agents and customers. Based on that massive amount of data, the AI can than inform agents how the conversation is likely progressing and prescribe what should be done next. Proponents of AI suggest that this technology helps mitigate some of the emotional labor, and thus burn out, by removing the decision-making responsibility away from the agent. However, in this case, one must also consider the importance of agency, autonomy, and discretion as a defense against work-related stress. That is, less autonomy, agency, and discretion that one has in one’s work usually relates to less engagement. This just further exacerbates an existing problem in this role as it removes elements of jobs that have shown to serve as resources against burnout-related behaviors. Companies will be well-served to consider that disruption may have both positive and negative consequences, and will have a profound impact on the drivers of performance in many roles.
There is speculation, of course, that ChatGPT and its AI peers will eliminate the need for certain jobs altogether. Computer programmer may seem especially vulnerable, as ChatGPT’s ability to generate code has received some attention early on. However much like the introduction of most technologies, the hype and future potential exceeds current application. Using the programmer role as an example, the capacity for ChatGPT to write code has not reached the level in which programming jobs will be in peril anytime soon. Right now, it can create only relatively simple programs. For example, asking ChatGPT to write code in Python to “scrape Champions League Football statistics from the internet” will get you close to a snippet of code that serves this purpose. However, ask it for something complex – such as a sophisticated game or business application, and it will admit its weakness and tell you that the task is currently beyond its abilities.
The problems it can manage will likely grow more nuanced and will be able to address more complex applications in the future, but we strongly believe that ChatGPT and similar language models will not make computer programmers obsolete anytime soon. While providing efficiencies in many aspects of the role, we will still require human oversight and understanding to ensure that the generated code is correct, efficient, and secure.
More importantly, computer programming and related technical roles are not only about writing code, they also involve designing and testing software, maintaining and updating existing systems, and most importantly, creatively problem-solving using the power of IT technology to solve new challenges, address new opportunities, and adapt to rapidly changing business environments. Moreover, computers are great at convergent thinking, while humans are strong at divergent thinking. ChatGPT is a tool that can be used to support programmers to speed up their development process and improve the quality of their code, it can’t replace the creativity, intuition, and problem-solving abilities of human programmers. By reducing cognitive load and time commitment by automating repetitive tasks, this type of technology frees the individual to focus on more strategic, complex, and creative aspects of the work. That is, the value of programmers will shift more toward being able to apply divergent thinking to creative problem-solving and reframing questions in ways in which computers and devices can be useful in solving increasingly complex business problems.
What Will Be Important to Know About Job Candidates and Employees Going Forward?
In the face of significant change, we must assess how all stakeholders interact with the technology and tools, how those tools might impact business challenges and opportunities, and how it changes the work that people do in impacted occupations. ChatGPT is just the latest in a series of technical advances that will continue to make convergent thinking work tasks much more efficient. While we are certainly not there yet, one can imagine the future of AI technology as the most efficient approach to problem-solving or decision-making in those situations that call for selecting from a universe of knowable options or involves analyzing and synthesizing information from various sources to come up with a single, correct answer or solution to a problem. This trend certainly did not start with ChatGPT. In 2017, McKinsey & Co. estimated that about 25% of work activities across all occupations could be automated by 2030. Those same data suggest that an estimated 60% of all occupations listed by the Bureau of Labor Statistics could see a third of all work activities automated over the coming decades. Computers are really good at logic, reasoning, and deductive thinking to arrive at a specific conclusion. We would be remiss not to let them do that.
So then, what are we looking for in those employees who are using the tools and leveraging the technology? To answer that question, we must consider the full context in which we assess the abilities of technology professionals. It starts with the ability to apply what one knows about the technology to solve increasingly complex business issues. Whether we are talking about rapid changes in technology, globalization, lower barriers to entry resulting in hyper-competition, a shift in customer expectations, or any of the other reasons that drive the constant need to adapt, companies across all industries will need to create cultures that emphasize idea generation and innovation. The good news is that technology like ChatGPT will continue to get better at handling repetitive, convergent tasks. The goal for the human element will be to think beyond the obvious and explore new and unexpected possibilities, which can lead to innovative solutions and breakthroughs. This coupled with the tendency to design systems and frame problems that can be solved via technology, will increasingly define the most successful companies in the information economy. Therefore, thinking style matters and the most relevant thinking style constructs to assess in technical candidates in the age of ChatGPT include:
- Abstract Reasoning: ability to recognize patterns in data and to see logical relationships. Taken together, these are strong predictors of learning portability; taking learning from one context and applying it to solve new problems or problems in novel contexts.
- Computational Thinking: Tendency to solve problems, design systems, and make decisions by drawing on the concepts fundamental to computer science. This involves algorithmic thinking
- Ideation: Generating and developing new ideas or concepts. It involves brainstorming, exploring, and experimenting with various approaches to solving problems, creating products, or developing strategies.
- Learning Agility: the willingness and ability to learn from experience and then apply those lessons to perform successfully under first-time, tough, or different conditions.
In the age of ChatGPT, we are less interested in assessing program knowledge via specific coding challenges and more interested in assessing the underlying cognitive traits and thinking styles that will generalize and help us predict performance in novel, ambiguous, and uncertain problem-solving situations. So while we will continue to support the assessment of programming knowledge via challenges, we strongly recommend that they be used as confirmatory measures.
ChatGPT and similar technologies arrive at a point in which innovation and creative problem-solving are more important than ever. As we know, there will be an increasing need for those in high-tech roles to collaborate with cross-functional teams that contain high-level professionals of differing backgrounds to solve problems or address opportunities that have not been addressed before. That is, high-tech professionals will need to communicate with people who, while very smart, are not experts in IT technology related to issues or types of problems that neither has dealt with before. This means that the technology professional must be able to both convey information in a clear, non-jargonish way, as well as be inquisitive and seek information in such a way as to draw from non-technical people all the business problems, use cases, and outcomes desired from the development of a solution. This increased need for collaboration among highly specialized professionals pushes some additional psychometric constructs to the forefront of candidate assessment. These include:
- Communicating: ability to communicate information in a concise, direct, and unambiguous way. The tendency to understand how the message content is received by others and make adjustments to style to ensure mutual understanding
- Collaborating: ability to work with cross-functional teams of highly specialized professionals to achieve solutions to complex business problems
- Seeking Information/inquisitiveness: Asking insightful questions to get a deeper understanding of issues. leveraging experts and other key individuals as important sources of information.
Finally, there are a set of intra-personal and self-regulation skills that make it more likely that an individual will perform well and thrive in technology roles.
- Resilience: working with technology can be challenging and complex and often requires sustained effort and attention to detail. Those that adapt to changing circumstances, persevere through difficult tasks, and maintain a positive attitude in the face of adversity perform at much higher levels.
- Self-Efficacy: Being innovative involves taking risks. Success in these roles will require experimenting with new ideas and solutions, and not being deterred by the possibility of failure.
- Taking Initiative: Success in dynamic business and technology environments requires one to take on new responsibility and challenges. There is an increasing need to be proactive in learning new programming languages, frameworks, and technologies, as well as keeping up with industry trends and best practices.
ChatGPT and The Reliability and Validity of Assessment Results
Reliability (consistency) and validity (accuracy) are the key characteristics of an employment assessment. That is, when we assess an individual using a pre-employment assessment, we want to know the degree to which that assessment made at one point in time will correlate with performance later (is performance on the test consistent with performance on the job?). Response distortion (“cheating”) via copy-and-paste from tools like ChatGPT and other forms of plagiarism are understandable concerns for many hiring managers. ChatGPT has made it easier for test-takers to submit code that is not their own, a situation that may call into question the reliability of assessments that rely on programming challenges as the primary means of assessment ability.
IKM’s adaptive testing methodology helps mitigate issues of response distortion, as no two administrations of the assessment are likely to produce the exact same items. Further, since IKM assessments more directly measure the cognitive constructs that generalize and drive performance across many technologies, they are much less susceptible to the purposeful response distortion that can be found in coding challenges.
We will continue to employ the current technologies (e.g., monitoring in-test computer behavior, video monitoring, plagiarism detection software to identify and flag any text that matches text from a source outside of the test taker’s own knowledge, etc.) available to mitigate the risk of this form of response. We are already taking these measures, and there is little reason to believe they will be stymied by ChatGPT. OpenAI itself is working on solutions, as they studying hiding cryptographic signals, called watermarks, in ChatGPT results, so that they’ll be more easily identifiable by anti-cheating software.
This brings us to the second point of validity (accuracy). That is, to what degree are we accurately measuring what we should be measuring? It is important to keep in mind that this (as well as any other emerging technologies) will likely be available to employees once they are hired. Therefore, one of IKM’s most important challenges is to assess test-takers’ ability to perform within the context of all the tools, technology, and job aids available to the job roles for which we assess. This is already reflected in the psychometric assessments we employ alongside knowledge tests, and will continue to be a bigger part of our offerings to best predict performance within the context of ever-changing technologies.
IKM will continue to lead the way in evolving high-tech employment testing to match the evolution of technology and its various applications.
About the author
Tom Schoenfelder, Ph.D. is an Industrial/Organizational Psychologist with extensive experience in employee assessment, talent development, applied research, and organizational consulting. He is IKM’s Vice President of Research and Development, leading the creation of technical and psychometric assessments tools used to predict performance and prescribe professional development in high-tech and specialized occupational roles.