Nitor Infotech's April Tech Bulletin

As part of Nitor Infotech, an Ascendion company, our engagement in innovative technological implementations & experimentations in AI has yielded valuable insights. We would like to illustrate how “Reinforcement Learning from Human Feedback”, RLHF, is transforming AI learning through human feedback. 

This edition, co-authored with Anil Khanna , Vice President, Technology Practices, at Nitor Infotech, aims to underscore the significant and compelling nature of RLHF and its transformative impact on creating relevant and more responsive solutions. 

Before we proceed, for anyone who isn't sure what the topic is: 

What is RLHF? 

RLHF stands for “Reinforcement Learning from Human Feedback”. It is a machine learning technique that trains AI models by incorporating human preferences and feedback. Instead of relying solely on predefined reward functions, models learn from direct human evaluations of their outputs. 

Humans provide feedback to AI, indicating preferred and undesirable outputs. 

Sample this:  Imagine that you are training a coffee-making AI assistant: 

First attempts: “Espresso with 5 sugar cubes + pickle juice” → chaos 

Human guidance: Rank outputs from “undrinkable” to “barista-quality” 

Outcome: Learns to suggest “oat milk latte with cinnamon” → near-to-perfect morning brew 

Why It Matters: Human-Centric AI 

RLHF allies AI with human values, ethics, and preferences. The result? Imagine more useful and reliable systems. 

It's crucial for applications where subjective judgments are vital, such as language models, content generation, and interactive AI. 

It allows for a more intuitive interaction with AI. 

Now, how do humans factor into this?  

The Human in the Middle: Enhancing AI with Feedback 

The Role of Humans in RLHF: 

Humans provide labeled data by ranking or rating AI-generated outputs. This feedback guides the AI to refine its behavior. 

Look at the process of human feedback in training AI models: 

The process of human feedback in training AI models 

• Model generates outputs. 

• Humans rank/rate outputs. 

• Feedback trains a "reward model". 

• Reinforcement learning optimizes AI's behavior. 

Here are a couple of examples to help you understand it better: 

• Chatbots become more humanlike through RLHF. 

• Large Language Models (LLMs) utilize RLHF to align output with human intent. 

Now it is time to turn to some alternative approaches and considerations. 

Alternative Approaches and Considerations 

Shared below is a list of alternative approaches and considerations: 

• Direct Preference Optimization (DPO): 

Instead of training a reward model, DPO directly optimizes the policy based on human preferences. This leads to potentially more stable and efficient training. 

It simplifies the RLHF process by removing the reward modeling step. 

• Constitutional AI: 

This approach uses a set of principles or "constitutions" to guide AI behavior. It reduces the need for extensive human labeling. 

It aims to create AI systems that are inherently aligned with desired values. 

• Human-in-the-Loop (HITL) Learning: 

HITL involves continuous interaction between humans and AI during the learning process. This allows for real-time feedback and adjustments. 

It is especially useful in complex or dynamic environments. 

• Active Learning: 

AI models strategically select the most informative data points for human labeling. This minimizes the amount of human effort required. 

This is very useful to reduce the amount of human feedback required. 

• Considerations: 

Each approach has its strengths and weaknesses. The best choice depends on the specific application and available resources. 

All said and done, combining multiple approaches can often lead to the most robust and effective AI alignment. 

Let’s head toward the horizon, which holds the promise of RLHF, yes – but certain pitfalls need our attention. 

The Future with RLHF and Alternatives 

Here is a list of pitfalls to keep an eye out for: 

• A trace of bias could be present in human feedback, including sampling bias. 

• There are questions about the scalability of human labeling. 

• Defining and quantifying human preferences could be a challenge including harmful preferences. 

• Evaluating complex outputs could be challenging for humans. 

• Misleading behavior, thus tricking humans to provide positive feedback. 

Despite these potential risks, we are quite certain that numbers have been and will be showing up to show us improved AI-generated text quality and user satisfaction. That’s not all – some statistics are likely to attest to the fine-tuning potential of models. 

What is changing or set to change in AI development, especially in 2025? 

• There is a sure shift toward human-centered AI. 

• We can look forward to the creation of more intuitive, engaging, and trustworthy AI. 

• There is likely to be an increased emphasis on human interaction. 

We say - embrace curiosity and ask the right set of questions.  

We would love for you to join the conversation and explore where AI takes us and, equally importantly, where we take it. Share your comments. 

Head over to our AI page to enhance your knowledge.  

Subscribe to our monthly Tech Bulletin for regular updates on similar informative content!  

Be a part of our blog newsletter community. Here, we handpick and collate our resources on technological advancements.  

Until next time, explore our tech blogs loaded with actionable insights. They will fuel your brainstorming sessions at work and your next innovation sprint!