Prompt engineering skills: A possible cure on new automation anxiety

Prompt engineering deals with improving interaction between humans and AI

According to investment bank Goldman Sachs, the latest generative Artificial Intelligence (AI) models like GPT-4 are likely to displace – not necessarily replace – around 300 million full-time jobs in the coming years. Prompt engineering is one of the skills these white-collar occupations may need to adapt to survive the next automation boom.

Prompt engineering essentially deals with improving the entire interaction between humans and AI. A prompt is a set of instructions given to a machine learning algorithm on which the AI models are designed and generate a specific output. Prompt engineering skills execute systematic designing and refining of prompts to achieve accurate outputs from huge data structures of large language models (LLMs).

As a result of this rapidly increasing popularity, several online training and education platforms have started offering courses related to new techniques in this area. Vinay Pradhan, Country Manager, India, and South Asia, Udemy said generative AI models and particularly ChatGPT continued to be in high demand across the globe.

“Over the past few months, we have seen a rapid increase in both the number of courses created by our instructors and the number of learners enrolled in courses related to this topic. We have seen more than 300,000 enrollments globally, with more than 58,000 Indians learning ChatGPT on Udemy in mid-March. We currently offer more than 375 courses on ChatGPT, an increase of 28.4 per cent over the prior two weeks. And additionally, there are more than 1,000 courses about generative AI which include at least one section/lesson on ChatGPT,” Pradhan said.

Udemy has already listed several courses for prompt engineering starting at Rs 445 for beginners.

“Looking ahead, we will also continue partnering with instructors to launch more immersive, hands-on learning experiences for professional learners worldwide and increase skills development opportunities in India through new learning modalities.”

The rapidly growing profession of prompt engineers caught up attention last month after San Francisco-based AI startup Anthropic offered up to $335,000 for a prompt engineer and librarian job. “It’s a hybrid between programming, instructing, and teaching,” the firm said in the job description. It added that this position was a bit hard to hire for, as the field of prompt engineering is arguably less than two years old.

A major application of prompt engineering is in the process of training chatbots that work on large language models like GPT-4. These models lack domain-specific expertise and may generate responses that are factually incorrect or out of context or even harmful to users.

Beerud Sheth, Co-founder and CEO of chatbot building platform Gupshup said prompt engineering skills can provide relevant instructions that ensure the responses are accurate and appropriate.

“At Gupshup, we don’t have a specific job role that requires only prompt engineering as a skill. That said, all AI engineers that we hire are expected to have prompt engineering skills, and this would fall under their semi-technical skill sets, along with others such as data annotation and data labeling. These AI engineers/prompt engineers are involved in building software that in turn creates accurate and relevant prompts that elicit particular responses from the chatbot which are then passed on to the user,” Sheth said.

He said that while hiring AI engineers, “We look at their ability to design and implement effective prompts and this requires a strong understanding of natural language processing (NLP) and machine learning concepts. Essentially, we assess the technical skills of candidates, including their proficiency in programming languages and NLP libraries and knowledge of ML algorithms.”

Back in India however, the demand for prompt engineer roles remains at a nascent stage, as per the staffing experts.

“The current active demand for prompt engineers is nearly negligible in India. The overall demand for the larger AI, ML & NLP skills cluster in India remains at the sub 15K average, but Prompt Engineering demand is not a visible contributor to the count. Further, we are not witnessing an emerging demand for prompt engineers from startups or product back-offices in India at their current stage of AI development,” said Kamal Karanth, the co-founder of specialist staffing Solutions Company Xpheno.

A fine-tuned prompt can unleash the higher potential of a platform like ChatGPT by making it capture a particular writing style, format, or tone. The language model can even throw out a text simulating an expert in any area if prompted systematically.

“The Prompt Engineer role has caught the attention due to the novelty of the role amidst the generative AI buzz. However, the hyper-specialized nature of this role does not require talent in high volumes at the early stages of AI development. The demand explosion, if any, would occur when the AI products get production-ready and go live,” Karanth said.