2021 Asian Conference on Innovation in Technology (ASIANCON)

Pune, India. Aug 28-29, 2021

College Enquiry Chatbot using Rasa Framework

Siddhant Meshram Namit Naik Megha VR
Department of Electronics & Department of Electronics & Department of Electronics &
Telecommunication Telecommunication Telecommunication
SIES Graduate School of Technology SIES Graduate School of Technology SIES Graduate School of Technology
Nerul, India Nerul, India Nerul, India
sunil.siddhant17@siesgst.ac.in namit.naik17@siesgst.ac.in megha.vr17@siesgst.ac.in
Tanmay More Shubhangi Kharche
Department of Electronics & Department of Electronics &
Telecommunication Telecommunication
SIES Graduate School of Technology SIES Graduate School of Technology
Nerul, India Nerul, India
tanmay.more17@siesgst.ac.in shubhangi.kharche@siesgst.ac.in

Abstract—The growth of technologies like Artificial Contextual AI Chatbot [5], [6]. It is known to have
2021 Asian Conference on Innovation in Technology (ASIANCON) | 978-1-7281-8402-9/21/$31.00 ©2021 IEEE | DOI: 10.1109/ASIANCON51346.2021.9544650

Intelligence (AI), Big Data & Internet of Things (IoT), etc. has significant effect on chatbots and contextual assistants. Rasa
marked many advancements in the technological world since makes the chatbot more dynamic as compared to the
the last decade. These technologies have a wide range of conventional chatbot as it can understand the user’s input and
applications. One such application is “Chatterbot or data over a wider domain as compared to the conventional
“Chatbot”. Chatbots are conversational AIs, which mimics the chatbot [5]. Since, it requires less dataset as compared to
human while conversing & eliminates the need of human by conventional chatbot the model will take less time to train
automating mundane tasks. In the study undertaken, we have itself, hence reducing the system complexity.
created a chatbot in education domain & it is named as
“College Enquiry Chatbot”. This chatbot is a web-based Rasa framework develops chat-bot with artificial
application that analyses and understands user’s queries and intelligence and has capability to train itself with very less
provides an instant and accurate response. Rasa technology is data. Rasa being open-sourced, gives user the scope to
used to construct this chatbot. It’s an open-source technology, customize things according to their needs. Such chatbots can
which uses its two main packages i.e., Rasa Core & Rasa have multiple integrations and can connect number of data
Natural Language Understanding (NLU) in order to build a sources at a time [7].
Contextual AI Chatbot. NLU is used to infer the intent and to
extract the necessary entities from user input & the Rasa Core
provides the output by building a probabilistic model with the II. LITERATURE SURVEY
help of Recurrent Neural Network (RNN). Evaluation of the For the study undertaken we reviewed some papers &
model is done by getting a confusion matrix and performance have presented them as follow: Introduction of pair of open-
measures like Precision, Accuracy & F1 Score which come out source python libraries which are Rasa NLU & Rasa Core,
to be 0.628, 0.725 and 0.669 respectively on average basis. This which can be used to build conversational agents is presented
chatbot’s accuracy, lack of dependability on human resources,
in this paper [8]. They curated these libraries so that any
24 x 7 accessibility and low maintenance creates various
person with minimal knowledge of programming can
opportunities for its implementation. This conversational agent
can not only be used in educational institutions but also in
develop a chatbot. They also demonstrated their ‘probability
places where enquiry becomes a tedious task. of selecting action’ part of Rasa Core by asking it some
questions. These libraries are still under active development
Keywords—Chatbot, AI, Rasa, Rasa NLU, Rasa Core, NLP, & support for reinforcement learning, making NLU robust to
Web Application, Enquiry Chatbot typos and slang, and supporting more languages are some of
the topics under development.
I. INTRODUCTION A chatbot is developed is developed by the authors D.
Chatbots are software-based applications with embedded Prakash [9] which collects data related to student’s
knowledge, that aids in stimulating conversations with attendance & marks. This chatbot was created using Rasa
human users via textual or auditory methods [1-3]. platform & user interface is developed using Django. This
Extending Applications of Chatbots to domains like helper bot is trained to answers queries pertaining to
education has become the need of the hour in today’s fast- student’s academic performance & attendance. This bot
growing era of technological world [4]. Today, institutes provides good accuracy with queries related to student’s
have started designing their own enquiry bots to automate marks & attendance, but the queries outside this aren’t
their conversations with the students. These chatbots act as a answered with great accuracy. The efficiency of this chatbot
campus guide by helping the users find an accurate response can be increased by training it using some complex data &
to their queries. feature like voice recognition can also be implemented.

This paper is based on College enquiry chatbot built In [10], a functional framework has been designed for a
using Rasa Framework, made with an objective of helping chatbot, which integrates Rasa NLU & neural networks.
students with their admission related queries. RASA is an Intent Recognition & entity extraction is implemented in this
open-source technology, that uses packages like RASA Core system. The system is developed using two different
& Rasa Natural Language Understanding (NLU) to build a methods, by Rasa NLU & Neural Networks (RNN). This

978-1-7281-8402-9/21/$31.00 ©2021 IEEE 1

orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on February 07,2025 at 04:33:51 UTC from IEEE Xplore. Restrictions ap
 developed system is able to learn on its own & answers some
questions on finance after experimental comparison &
validation. According to paper, the Rasa NLU has more
accuracy while neural networks have better integrity to
classify entities from segmented words.
An analytical study & review of Rasa – An open-source
technology for creating conversational AIs is presented in
this paper [5]. The authors talk about some recent
developments in Rasa platform. The introduction of
reinforcement neural networks, creation of robust NLU, etc.
has been presented in this study. According to [1], these
topics were under development at that time. This study tells
that, Rasa platform is much better as compared to any other
open-source alternative as it has all the capabilities.
After reviewing all papers, we observed that the papers
suggested that, Rasa is an excellent open-source technology Fig. 2. NLU Training Data
which can be used to perform highly complex tasks. Hence,
for implementation of our College Enquiry Chatbot, we x To define the NLU Model Configuration: Rasa
decided to use Rasa. NLU comprises of different components which
together make up the pipeline structure, once we
III. RESEARCH METHODOLOGY have the training data in our hand, we can give it to
the NLU model pipeline (refer Fig. 3), all the
components that are present in pipeline would be
trained sequentially.

Fig. 3. NLU Pipeline

x Training NLU model: We have to train our model

again to recognize user input so that our bot is able
to understand intent behind the message.
Fig. 1. Rasa Framework [7]

A. To Install Rasa and Its Dependencies

We will require Rasa Core, Rasa NLU and spaCy
English language model. For running our chatbot on virtual
environment we would be requiring Anaconda for python
which provides simplified package management and
deployment. Fig. 4. Confidence Level

B. Teaching bot to recognize user input using Rasa NLU

C. Teaching chatbot to respond using Rasa Core
NLU is responsible for teaching a bot to recognize user
The next task is to make the bot to respond to messages
input [11], to achieve this we would create a Rasa NLU
by training the dialogue management model of Rasa core [6].
model and feed the training data that user has prepared. The
model will then convert this user data into structured format x Writing Stories: Dialogue management model uses
containing intents and entities. stories as its training data. A story is actually a
x Preparing NLU Training Data: The training data section of conversation that takes place between a
would comprise of a set of messages that user user and a chatbot (refer Fig. 5). The user inputs
expects to receive from bot (as shown in Fig. 2). are expressed in form of intents and entities while
This data is then linked with intents and entities chatbot responses are expressed as actions.
which Rasa NLU should learn to excerpt.

2
orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on February 07,2025 at 04:33:51 UTC from IEEE Xplore. Restrictions ap
 Fig. 5. Stories

x Defining Domain: The domain is like an entire

universe within which chatbot lives and operates, it
contains everything what a chatbot should know
(refer Fig. 6 & 7). It contains all the inputs it
should expect to get from the user, the actions it
should be capable of predicting, its responses and
which information to store.

Fig. 9. <script> tag of HTML

Fig. 10. Rasa-webchat Widget

Fig. 6. Domain-Intents

IV. RESULT

A. Data Validation
Validating our training data makes sure that there are no
errors or major inconsistencies in our domain, NLU, or story
data. If data validation shows any kind of error, training the
model will also result in a failure or it may yield a bad
performance, so it’s a good practice to run this check before
Fig. 7. Domain-Responses training our model. For our domain the data validation does
not result in any kind of conflicts or inconsistencies in NLU
D. Deploying the chatbot using socketio and Story data (as shown in Fig. 11), hence we are ready to
Firstly, we create a webpage where we would like to run train our final model.
our chatbot using basic html and css code, then we would
configure the credentials.yml file (as shown in Fig. 8) to B. Confidence Level
activate the socketio channel by defining its credentials i.e., Rasa uses this technique of classifying the intent behind
the user and bot message events along with the session the message based on the confidence level rank i.e., the
persistence status. Then we need to include a <script> tag (as intent which gets the highest confidence level (on a scale of 0
shown in Fig. 9) in our html file of our web page, we need to to1) is ranked first for that particular message and is chosen
call an api inside this script tag to include Rasa-webchat to be the predicted intent, where 0 is for least confident value
which is a feature-rich chat widget to deploy virtual and 1 is for most confident value. The confidence score
assistants made with Rasa on any website. Then we need to entirely depends on the type of training data that we have
run our local Rasa server and pass on the port number of this provided in the NLU dataset i.e., larger the training dataset
Rasa server to socket URL section inside the script tag so provided to each intent, the more confidently the model
that our Rasa server gets connected to socketio endpoint each would be able to identify or classify the said intent from the
time we open our webpage. Finally, the chat bot appears on user’s message with greater accuracy. For our model we are
our web page and we are ready to chat with our bot (refer getting an average confidence score between 0.98-0.99
Fig. 10). which clearly states that

Fig. 8. Credentials.yml

3
orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on February 07,2025 at 04:33:51 UTC from IEEE Xplore. Restrictions ap
 Fig. 11. Data Validation

TABLE I. CONFIDENCE LEVEL BASED ON NLU TRAINING DATA

Sr. No User’s Message Predicted Intent Confidence
1 “hello” greet 0.9993215799331665
2 “see you” goodbye 0.9782857894897461
3 “yeah” affirm 0.9997299909591675
4 “nope” deny 0.9988703727722168
5 “thanks a lot” thankyou 0.9906532168388367
6 “are you a bot?” bot_challenge 0.9900264143943787
7 “that is not what i am looking for” out_of_scope 0.7692297697067261
8 “provide me the link to college website” ask_college_website 0.9974669814109802
9 “where is sies gst located?” ask_college_location 0.9994360208511353
10 “how good is SIES GST?” ask_college_grade 0.9957651495933533
11 “how to contact college officials?” ask_college_contact 0.9978607296943665
12 “how good is the infrastructure of SIES?” ask_college_infrastructure 0.9995036721229553
13 “eligibility criteria for admission” ask_college_eligibility_criteria 0.9764007925987244
14 “documents required during admission” ask_college_documents 0.9991568922996521
15 “provide me the DTE code of SIES” ask_college_DTE_code 0.9948717951774597
16 “when will the admission process start?” ask_admission_schedule 0.9988873600959778
17 “how to apply through CAP?” ask_application_procedure 0.771625280380249
18 “what is the cutoff score for EXTC?” ask_cutoff_score 0.9849044680595398
19 “what is the intake capacity per branch?” ask_branches_intake 0.9902501106262207
20 “how many seats are reserved for tfws?” ask_seat_distribution 0.9944698810577393
21 “provide me the fee particulars” ask_fee_structure 0.9969478249549866
22 “how are the campus placements?” ask_college_placements 0.9999663829803467
23 “who can apply through institute quota?” ask_institute_quota 0.9898166656494141
24 “what does a minority institution mean?” ask_linguistic_minority 0.9072004556655884
25 “can I opt for TFWS seat?” ask_TFWS 0.9978354573249817
26 “what is the attendance criteria?” ask_college_attendence 0.9905030727386475
27 “what are the different student chapters?” ask_college_extracurricular 0.9998523592948914
28 “how good is the teaching faculty?” ask_college_faculty 0.9951142072677612
29 “how good are the academic results?” ask_college_results 0.940822958946228
30 “important links regarding admission” ask_important_links 0.9998020529747009
31 “what are the office timings?” ask_college_timings 0.9943576455116272
32 “does college have canteen facility?” ask_yes 0.9999825358390808
33 “is the college autonomous?” ask_no 0.9945077896118164
34 “who made you?” ask_creators 0.9986181259155273

Fig. 12. Report Log (NLU & Core Test)

4
orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on February 07,2025 at 04:33:51 UTC from IEEE Xplore. Restrictions ap
 Fig. 13. Intent Confusion Matrix

the model is confident enough to predict the said intent from the slots being filled up diagonally, which signifies that the
the user’s message (refer Table. 1). model has a true prediction for that particular intent. Similar
is the case for our trained model which shows that our model
C. NLU & Core Test has predicted all the intents accurately without any confusion
Using “rasa test” command inside the CLI gives us the and the diagonal values represent the total number of
provision to run the tests on NLU and Core data of our samples that it has predicted correctly for that particular
model and helps us to visualize our trained model in the form intent. The intent histogram in Fig. 15 allows us to visualize
of confusion matrix and histograms. The intent confusion the confidence level against all the predictions from the
matrix in Fig. 13 shows us that which intents are mis- training samples. The correct and incorrect predictions are
understood for others, if all the intents are mapped or shown by blue and red bars respectively.
predicted correctly then the matrix would have resulted in all

5
orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on February 07,2025 at 04:33:51 UTC from IEEE Xplore. Restrictions ap
 Fig. 14. Story Confusion Matrix

Improving the quality of our training data will move the

blue histogram bars up the plot and the red histogram bars
down the plot. It can also help in reducing the number of red
histogram bars itself. For our trained model we are not
getting any misses (red bars) but only hits (blue bars) by
considering the predictions from around 488 training
samples, with all the samples being predicted with a
confidence score of approximately 0.96-0.97.
The story confusion matrix (refer Fig. 14) which is based
on our trained Core model (consisting of stories and
utterances) shows that some of the utterances are mistaken
for others due to the fact that some of the slots are being
filled up in the non-diagonal locations.
The report log (refer Fig. 12) clearly enlists some of the
parameters of our model like Precision, Accuracy and F1-
Score on average basis, which are 0.628, 0.725 and 0.669
respectively.

Fig. 15. Intent Histogram (NLU & Core Test)

6
orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on February 07,2025 at 04:33:51 UTC from IEEE Xplore. Restrictions ap
 Fig. 16. Report Log (Testing Data)

Fig. 17. Report Log (Cross Validation)

D. Testing Data E. Cross Validation

In addition to just testing our model on seen data we can Rasa also provides us the provision to test our model
also test it on an unseen data. Once our assistant gets more extensively using cross-validation by creating multiple
deployed on a real physical environment it will be open to all train and test splits in all possible combinations. The intent
sorts of messages that it hasn’t seen in its training data, so to histogram in Fig. 19 makes it clear that testing our model
simulate this scenario we have to set aside some part of our extensively results in inappropriate mapping of intents with
data for testing. Rasa gives us the provision to split our data some of the slots being filled up in non-diagonal positions
into train and test sets, it creates an 80/20 split of our this indicates that the model mistakes these intents with
train/test set by default. We can see that how well our trained others, also the intent histogram shows a uniform distribution
NLU model predicts the data from the test set that we have of hits and misses on 484 testing samples, with majority of
generated. From the report log (refer Fig. 16) it is clear that the samples being predicted with a confidence score of
our trained model is able to classify all the intents accurately approximately 0.96-0.97.
and is visualized from the intent confusion matrix as well
with all the slots being filled up diagonally. The intent
histogram in Fig. 18 shows the distribution with all the hits
and zero misses on 98 testing samples, with all the samples
predicted with a confidence score of approximately 0.96-
0.97.

Fig. 19. Intent Histogram (Cross Validation)

The report log (refer Fig. 17) clearly shows a descend in

train to test parameter values where Train to Test, Accuracy
value fall from 0.999 to 0.700, Precision value fall from
1.000 to 0.712 and F1-Score value fall from 0.999 to 0.679.
Fig. 18. Intent Histogram (Testing Data)

7
orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on February 07,2025 at 04:33:51 UTC from IEEE Xplore. Restrictions ap
 V. CONCLUSION [3] K. Deepika, V. Tilekya, J. Mamatha and T. Subetha, "Jollity Chatbot-
A contextual AI Assistant," 2020 Third International Conference on
The College Enquiry Chatbot, developed using Rasa, Smart Systems and Inventive Technology (ICSSIT), 2020, pp. 1196-
which being an open-sourced framework gives great 1200, doi: 10.1109/ICSSIT48917.2020.9214076.
functionalities and NLP toolbox through its Rasa stack. By [4] H. Gawade, V. Patil, P. Vishe and S. Kolpe, “College Enquiry Chat-
easing the task of deployment and server creation, it helps to Bot System,” International Journal of Engineering Research &
Technology, vol. 9, no. 9, pp. 741-742, September 2020, doi:
focus, improve and add various functionalities in Chatbot, so http://dx.doi.org/10.17577/IJERTV9IS090396
that it becomes complete and more intuitive to use. Using
[5] M. Joshi and R. K. Sharma, “An Analytical Study and Review of
appropriate set up, we can very well do the job of intent Open Source Chatbot Framework,” International Journal of
extraction and keep the user involved in the conversation by Engineering Research & Technology (IJERT), vol. 09, no. 06, June
the dialogue management even with less data. Using this 2020. [Online]. Available: https://www.ijert.org/an-analytical-study-
chatbot will not only benefit the students to solve their query and-review-of-open-source-chatbot-framework-rasa
but also the institution to offer courses in demand, know [6] J. Shabbir, M. U. Arshad and W. Shahzad, “NUBOT: Embedded
current trends and manage intake seats accordingly. Addition Knowledge Graph With RASA Framework for Generating Semantic
Intents Responses in Roman Urdu,” February 2021. [Online].
of new questionnaires after failure of chatbot to respond it Available: https://arxiv.org/pdf/2102.10410.pdf
correctly will ensure it updating in database and can answer [7] A. Jiao, “An Intelligent Chatbot System Based on Entity Extraction
it through notification or as and when asked in future. Using RASA NLU and Neural Network,” IOP Conf. Series: Journal
Moreover, a problem rarely faced by chatbot with a ‘person of Physics: Conf. Series, 2020, doi:10.1088/1742-
with disability’ (PwD) can choose to type or speak according 6596/1487/1/012014
to their dormant ability with the addition of feature such as [8] T. Bocklisch, J. Faulkner, N. Pawlowski and A. Nichol, “Rasa: Open
Voice recognition. Many a times aspirants which are from Source Language Understanding and Dialogue Management,”
December 2017. [Online]. Available:
different parts of country and don’t understand English, https://arxiv.org/abs/1712.05181
Multi lingual feature can be introduced. Hence this chatbot [9] D. Prakash, B. Mahesh, H. N. Mohammed Azharuddin, G. Namratha
could find its place in many institutions because of its and J. Madhura, “Chatbot in Education System,” International Journal
accuracy, lack of dependability on human resources, 24x7 of Engineering Science and Computing, vol. 9, no. 5, pp. 21926-
accessibility and low maintenance. 21929, May 2019.
[10] K. Bhaskar, “Conversational AI chatbot using Rasa NLU & Rasa
Core: How Dialogue Handling with Rasa Core can use LSTM by
REFERENCES using Supervised and Reinforcement Learning Algorithm,” Medium,
[1] S. Pawar, O. Rane, O. Wankhade and P. Mehta, “A Web Based Accessed: 10 May, 2021. [Online]. Available:
College Enquiry Chatbot with Results,” International Journal of https://medium.com/@BhashkarKunal/conversational-ai-chatbot-
Innovative Research in Science, Engineering and Technology, vol. 7, using-rasa-nlu-rasa-core-how-dialogue-handling-with-rasa-core-can-
no. 4, pp. 3874-3880, April 2018. use-331e7024f733.
[2] S. Punith, B. Chaitra, V. Kotagi and R. M. Chethana, “Research Paper [11] Abdellatif, K. Badran, D. Costa and E. Shihab, "A Comparison of
on Chatbot for Student Admission Enquiry,” Journal of Advancement Natural Language Understanding Platforms for Chatbots in Software
in Software Engineering and Testing”, vol. 3, no. 1, pp. 1-9, 2020. Engineering," in IEEE Transactions on Software Engineering, doi:
10.1109/TSE.2021.3078384.

8
orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on February 07,2025 at 04:33:51 UTC from IEEE Xplore. Restrictions ap