2024 10th International Conference on Advanced Computing and Communication Systems (ICACCS)

Automated Business Report Summarization Using

Transformer Model
2024 10th International Conference on Advanced Computing and Communication Systems (ICACCS) | 979-8-3503-8436-9/24/$31.00 ©2024 IEEE | DOI: 10.1109/ICACCS60874.2024.10716930

Faizal B Sajimon Abraham Sijo Thomas

School of Computer Sciences, School of Computer Sciences, School of Computer Sciences,
Mahatma Gandhi University, Mahatma Gandhi University, Mahatma Gandhi University,
Kottayam, Kerala, India Kottayam, Kerala, India Kottayam, Kerala, India
[email protected] [email protected] [email protected]

Abstract—Text summarization is still s demanding procedure succinct summaries. Numerous text formats, such as articles,
in various core domains such as medical, legal and finance. Pro- business documents, news reports, and more, are supported by
duce a concise summary from a given data is always a Herculean
these platforms.
task. However, the recent trends in artificial intelligence pave a
way to some methodologies that can leverage the effort to a great A crucial component of data analysis and decision-making
extent. So, summarization is still a topic of discussion in various in businesses is the summarization of business data. In order
domains. When considering financial data, most of the data relies to make corporate data more readable and useful, it is
on visual representations like tables. In this situation, necessary to extract important trends and insights from huge
summarization of business report is a challenging task. This paper datasets. Business data summary covers a wide range of data-
introduces a methodology for handling tabular data and text data
together for abstractive summarization with help of transformer driven decision support applications. Financial analysis,
based models. Here, an abstractive summarization model is made Customer Relationship Management (CRM), Business
and the effectiveness of the model is tested. Intelligence Dashboards, Financial Forecasting are some
key areas in business field that summarization has its own
Keywords—Abstractive text summarization, Rouge met- rics, role. Business reports are rich with table representations
Encoder–decoder, Self-attention mechanism, Transformer ar
chitecture. since they provide an ordered and structured means of
I. INTRODUCTION presenting and analyzing data. Tables offer a structured and
logical manner of displaying data, which facilitates readers’
Natural Language Processing (NLP) has inevitable role comprehension of detailed data. Tables give data a precise,
in every domains that dealt with text data.NLP has a wide quantitative representation that makes analysis easier. This is
range of application areas, driven by its ability to bridge the particularly significant in the corporate world, where accuracy
gap between human language and computer understanding. is essential for processes like performance reviews, financial
NLP involves a variety of tasks, including natural language reporting, and other vital decision-making. Many financial
understanding, sentiment analysis, language translation, and regulatory authorities still follows the method of reporting the
text summarization. A vital component of natural language data in specified intervals. Tables offer an organized and
processing (NLP) is text summarization, which tackles the transparent means of presenting pertinent data, which helps to
problem of extracting important information from numerous convey relevant information.
sources of text while maintaining its fundamental meaning. Text summarization regarding business report has a
Effective text summary is becoming more and more necessary uniqueness since the structure of the financial report is
as the amount of digital information keeps growing different from other formats and contents of the report is
exponentially. NLP uses language models and algorithms to mainly figures and tables which convey the information in
find and extract the most important information from a a precise way. So the task of summarization of business data
document, which is a key component in automating this is always a challenging task. The task of summarization can
process. be summed up to following processes.
NLP techniques can be broadly divided into two categories
when it comes to text summarization: extractive and A. Objectives of the Study
abstractive methods. Whereas abstractive summary creates The main motivation regarding the proposed methodology
fresh, succinct words that communicate the major concepts is to find a concise summary for business reports that should
in a more humane way, extractive summarization chooses and consider the data from tables and textual information. The
puts together important sentences or phrases straight from the summarization methods including extractive and abstractive
source text. methods are now trending, however, extractive methods [3]
Advanced language models [1], machine learning have some limitations when handling longer text as input. On
algorithms, and linguistic analysis are the key techniques the other hand, Nuanced understanding is critical in highly
used by NLP-powered summarizing systems to comprehend
thecontext, extract crucial information, and provide clear and

979-8-3503-8436-9/24/$31.00©2024 IEEE

254
Authorized licensed use limited to: JNT University Kakinada. Downloaded on December 16,2024 at 10:50:11 UTC from IEEE Xplore. Restrictions apply.
 2024 10th International Conference on Advanced Computing and Communication Systems (ICACCS)

specialised or technical topics, where abstractive Furthermore, the suggested method outperforms a
summarization techniques may not perform effectively. They traditional Transformer with the self-attention mechanism
might omit crucial technical specifics or generalise since it makes use of a variety of expert processes from the
information. The main objectives can be summarized as Switch Trans- former to help capture a variety of patterns.
follows.
• Text summarization [7] has advanced significantly thanks III. THE SUMMARIZATION APPROACH
to transformers, in particular large pre-trained models Based on the discussions in various research papers, the basic
such as GPT (Generative Pre-trained Transformer) and approach to text summarization can be summarized asin Fig
BERT (Bidirectional Encoder Representations from 1 [1].
Transformers).Additionally, Transformers [2] have the
capability to be tailored for specific domains or
specialized fields by adjusting them using data that is
specific to those domains. This adaptability is crucial for
various sectors such as finance, low and others. So,
develop a summarization model for business data based
on transformers that can accurately summarize it.
• A comparative study over the performance of the model
against various measures such as ROUGE (Recall-
Oriented Understudy for Gisting Evaluation) score and
F measure.
II. RELATED STUDY
The majority of research in this field focuses on the process
of creating and preparing summaries. Since the abstractive
method requires more NLP support, the majority of researchers
choose the extractive method. Anushka Gupta et al worked on
the paper titled Automated News Summarization Using Trans-
formers [5] described about various extractive summarization
models based on transformer architecture. This research
presents a thorough evaluation of many pretrained models for
Fig. 1. Summarization approach.
text summarization that are based on transformer architecture.
We have analysed and compared the summaries produced A. Summarization based on algorithms
by machine learning models using the BBC news dataset,
which includes text data suitable for summarization as well as Supervised algorithms generates text summary by teaching
human-generated summaries. Here they used CNN/Daily mail the system using a labeled data set. Each piece of content has
dataset and the comparison was made on different matrices. its matching summary. The program learns which sentences
Mohammad Bani-Almarjeh and Mohamad-Bassam Kurdy or words matter the most from features noted during teaching.
together worked on an Arabic abstractive text summarization For supervised summarization tasks, transformer concepts like
using RNN-based and transformer-based architectures [6]. as BERT (Bidirectional Encoder Representations from
They have evaluated and compared the efficacy of several Transformers) or RoBERTa can be optimized. Based on the
pre-trained language models, such as AraBERT, AraBERT, input document and summary pairings, the model obtains the
AraGPT2, and AraT5, and model architectures, such as RNN- ability to produce summaries.
based and Transformer-based ones, for Arabic abstractive In unsupervised text summarization, the system retrieves
summarization. They have also tested the system with ROUGE important information from the text Without using labeled
metrics and manual human evaluation methods. Additionally, training data. To find significant content, these techniques
the system was fine-tuned with encoder-only Transformer frequently make use of clustering, graph-based, or statistical
models using a BERT2BERT-based encoder-decoder techniques. The availability of labeled training data, the
architecture. required degree of control over the summarization process,
Another work from Rahali, Abir and Akhloufi, Moulay A and the particular objectives of the domain are generally
[12] They suggest a streamlined Switch Transformer structure the deciding factors when selecting between supervised and
and use the CHU Sainte-Justine hospital’s tiny clinical text unsupervised techniques.
classification dataset to train it from scratch. The simpli- B. Available Datasets for Summarization
fied small-scale Transformer models perform better than pre-
trained BERT-based models, such as DistillBERT, Camem- • DUC (Document Understanding Conference) Datasets.
BERT, FlauBERT, and FrALBERT, according to our results. Datasets from DUC’s yearly summarizing contests are

255
Authorized licensed use limited to: JNT University Kakinada. Downloaded on December 16,2024 at 10:50:11 UTC from IEEE Xplore. Restrictions apply.
 2024 10th International Conference on Advanced Computing and Communication Systems (ICACCS)

made available. Usually, these databases contain In order to take use of the sequence order, an additional
extraction summaries that were created by humans in layer of positional encoding is layered between the encoder
addition to news items. and decoder stacks in models that lack recurrences or
• CNN/Daily Mail Dataset This dataset consists of news convolutions. An encoder is made up of N layers, each of
articles paired with multiple extractive reference sum- which has two sublayers. A multi-head mechanism develops
maries. It is widely used for extractive summarization self- attention on the first sublayer. It’s been shown that the
research. The CNN/Daily Mail dataset also provides multiple- head mechanism produces the outputs in parallel by
abstractive summaries. It is a popular choice for both taking a (different) linear projection of the queries, keys, and
extractive and abstractive summarization tasks. values to create the final output. A fully connected feed-
• Datasets for Financial News:Financial News Dataset forward network with two linear transformations and
from Reuters: Reuters offers a variety of financial news activation of Rectified Linear Units (ReLUs) is present in the
datasets covering equities, bonds, commodities, and second sublayer.
market movements, among other subjects. These datasets The decoder stack in the first sublayer applies self-attention
frequently have summaries that were created by humans. across the heads, receives the previous output, and provides
• Bloomberg News Dataset: For reliable financial news, positional information. Unlike encoders, which focus on every
turn to Bloomberg. For summaries of financial news word regardless of order, decoders only pay attention to the
articles, you can search through their archives. words that come before them. Because of this, the word at
• Financial Phrase Bank: This dataset was created position i can be predicted based only on the words that come
especially for sentiment analysis in the financial industry. before it in the sequence. A mask is superimposed over the
It does not directly provide summaries, but it does include values obtained by multiplying Q and K by a scale factor in
sentiment- labeled financial words that can be used to a multi-head attention mechanism, which is the execution of
train abstractive summarizing algorithms. multiple, single attention functions in parallel.
IV. EXISTING APPROACH FOR TEXT SUMMARIZATION Transformer architectures depend heavily on attention
USING TRANSFORMER ARCHITECTURE mechanism, which is essential for extracting contextual in-
formation from input sequences. Since its introduction in the
Transformers’ attention mechanism is derived from the ”Attention is All You Need” paper by Vaswani et al. [4], the
encoder-decoder architecture included in RNNs. It can per- attention mechanism has grown to be an essential component
form tasks involving sequences to sequences (seq2seq) while of many tasks involving natural language processing.
eliminating the sequential element. A Transformer can process
data more quickly and with higher parallelization than an RNN • Query: here we define the parameters that need attention
because it doesn’t process it in sequential order. As shown in during the process.
• Keys: An encapsulated form of input elements and keys
together which features vectors. The key assignment
process should be careful since this will eventually help
us to smartly select attention-required elements based on
thegiven query. Values: Besides input elements, there are
value vectors as well. Our goal is to create an average of
these feature vectors.
• Score function: The score function takes input as key
and the query which delivers the output as corresponding
query-key’s weight or attention score

V. PROPOSED METHODOLOGY FOR REPORT

SUMMARIZATION
The study proposing architecture based on transformer
model [9]. This is a sequence based text summarizer that takes
input as a text and produce a summary of the given long text.
As discussed about transformer model, here text summarizer
acts as a pivotal element in the transformer architecture.
At first, the dataset is taken and do some base on the nature
Fig. 2. Transformer model overall architecture. of the data. After a word embedding stage, the vector
representation of the word can be given to a transformer
Fig 2 [3] there are two main components are available in this
model. The model will deliver an output summary with the
architecture. An encoder stacks — Nx identical encoder layers.
help of a Natural Language Engine (NLG).
A decoder stacks — Nx identical decoders layers.

256
Authorized licensed use limited to: JNT University Kakinada. Downloaded on December 16,2024 at 10:50:11 UTC from IEEE Xplore. Restrictions apply.
 2024 10th International Conference on Advanced Computing and Communication Systems (ICACCS)

Then the output of the model can be compared with the in comprehending the input text’s semantics and context.
reference summary. The performance evaluation can be done This is especially crucial for abstractive summarization, where
with different metricslike ROUGE. the model must provide succinct, logical summaries that
accurately convey the original content’s meaning. Tokenizing
the input text into meaningful units, or tokens, and then
embedding those tokens into vector representations are the
steps involved in NLU. This phase guarantees the preservation
of word contextual links while giving the transformer model
an organised input format.
NLU uses Named Entity Recognition (NER) that aid in
identifying and comprehending textual entities and references
include co reference resolution and NER. Since the model
must collect and represent important items and preserve
coherence between pronouns and their antecedents in the
summary, this information is useful for summarising.
Transformers use attention techniques [13] to give various
input sequence segments varying weights. The attention
mechanism is directed by NLU to concentrate on pertinent
words or phrases that are essential for summarization. This is
necessary to comprehend the source content and produce
Fig. 3. Architecture for proposed methodology
insightful summaries. The Natural Language Generation
(NLG) [8] module will create the summary and then the
The core part of the proposed module is input data. Unlike
generated summary will be evaluated based on the available
the traditional text data, here the input data is a blend of tabular
metrics such as ROUGE. The main emphasis of ROUGE
data and the text data. So, an additional tabular data extraction
metrics is the measure of overlap between the n- grams
will be added to the entire system and then the system will
contiguous sequences of n elements, often words or
receive input as data from table and text. The aim of this work
characters. The main objective is to evaluate the generated
is to create a benchmark model for text summarizer and then
text’s recall and accuracy in relation to the reference text.
based on this transformer summarizer model we will change
the input module. So, the input module will be capable of Precision: The proportion of n-grams produced by the sys-
accepting the data from tables and then NLG module will tem that are also included in the reference summary. A better
properly create sentences based on the keys generated from the precision [11] means that pertinent information is included in
table. Module B in fig.3 will act as a benchmark methodology the system summary.
to accept the input and module A will develop input text from Recall: The proportion of reference n-grams included in the
the business report [10]. So, the entire system will be capable summary produced by the system. Greater recall [11] suggests
of handling both test and tabular data for as input for the that significant information from the reference is captured by
summarization the system summary.
F1 Score: A balanced metric produced by taking the
A. Module A harmonic mean of recall and precision. The F1 score is
Module A will be act as the input generation section frequently employed to provide an overall performance
that collect data from tables and then generate sentences summary [11].
for the data corresponding to the table row and columns.
Example, we have a column profit based on regions then we C. Result Analysis
analyze the profit column and then generate proper instance
as ’Maximum profit was from southern region during the third
quarter of 2023”. In this way table data will be analysed and
the generated input will combine with the text data also.The
sentence generation can be done with NLG by considering the
tokens from table ¡col, row¿ value.
B. Module B
Module B of the proposed architecture is the benchmark
model that process the input from module A. Transformer-
based text summarization relies heavily on Natural Language
Understanding (NLU) to help the model understand and
process the incoming text efficiently. NLU aids transformers Fig. 4. Accuracy graph

257
Authorized licensed use limited to: JNT University Kakinada. Downloaded on December 16,2024 at 10:50:11 UTC from IEEE Xplore. Restrictions apply.
 2024 10th International Conference on Advanced Computing and Communication Systems (ICACCS)

The model was evaluated based on the generated summary

and the existing one by analysing ROUGE score and the
model produced a score of 48.5 for Rouge-1. The model was
trained on 40 epochs. This research work is an ongoing work
where only text summarization work was done to find the
effectiveness of the system. Additionally, the main part of the
research such as designing input module to handle tabular data
for summarization is still under progress.
VI. CONCLUSION
In this paper, we proposed a novel architecture for business
report summarization using transformer method. The study
proposed an NLP based transformer architecture for automatic
text summarization and a methodology for changing the input
sequence to a complex module that can handle tabular data
as input. On completion of this study, this is evident that
the system can be useful in various domains such as business
intelligence and stock market analysis. The model performance
can be enhanced by fine tuning the system with more accurate
measures. Only ATS is included in this study. To further
enhance the suggested model’s performance, a few problems
must yet be resolved.
REFERENCES
[1] F. B and S. Abraham, ”NLP Based Automated Business Report Sum-
marization,” in Proceedings of the 2022 International Conference on
Innovative Trends in Information Technology (ICITIIT), Kottayam,
India, 2022, pp. 1-4. DOI: 10.1109/ICITIIT54346.2022.9744151.
[2] K. T. Chitty-Venkata, M. Emani, V. Vishwanath, and A. K. So-
mani, ”Neural Architecture Search for Transformers: A Survey,” in
IEEE Access, vol. 10, pp. 108374-108412, 2022. DOI: 10.1109/AC-
CESS.2022.3212767.
[3] Lin, T., Wang, Y., Liu, X. Qiu, X. (2021). A Survey of Transformers
(cite arxiv:2106.04554)
[4] A. Vaswani et al., ”Attention is All you Need,” in Proceedings of the
Neural Information Processing Systems (NIPS), 2017.
[5] A. Gupta et al., ”Automated News Summarization Using Transformers,”
ArXiv, 2108.01064, 2021.
[6] M. Al-Maleh and S. Desouki, ”Arabic text summarization using deep
learning approach,” Journal of Big Data, vol. 7, pp. 1-10, 2020.
[7] S. Bhatia, ”A Comparative Study of Opinion Summarization Tech-
niques,” in IEEE Transactions on Computational Social Systems, vol.
8, no. 1, pp. 110-117, Feb. 2021. DOI: 10.1109/TCSS.2020.3033810.
[8] F. Zhang, G. An, and Q. Ruan, ”Transformer-based Natural Language
Understanding and Generation,” in Proceedings of the 2022 IEEE
International Conference on Signal Processing (ICSP), pp. 281-284,
2022. DOI: 10.1109/ICSP56322.2022.9965301.
[9] S. Pasch and D. Ehnes, ”NLP for Responsible Finance: Fine-Tuning
Transformer-Based Models for ESG,” in Proceedings of the 2022 IEEE
International Conference on Big Data (Big Data), Osaka, Japan, 2022,
pp. 3532-3536. DOI: 10.1109/BigData55660.2022.10020755.
[10] H. Koh, J. Ju, M. Liu, and S. Pan, ”An Empirical Survey on Long Docu-
ment Summarization: Datasets, Models and Metrics,” ACM Computing
Surveys, vol. 55, 2022. DOI: 10.1145/3545176.
[11] H. S., A. S., A. V., and R. K. Grace, ”Summarization of News Articles
Using Transformers,” in Proceedings of the 2022 5th International
Conference on Advances in Science and Technology (ICAST), Mumbai,
India, 2022, pp. 159-163. DOI: 10.1109/ICAST55766.2022.10039608.
[12] A. Gillioz et al., ”Overview of the Transformer-based Models for NLP
Tasks,” in Proceedings of the 2020 15th Conference on Computer
Science and Information Systems (FedCSIS), 2020, pp. 179-183.
[13] P. N. Singh and S. Behera, ”The Transformers’ Ability to Implement
for Solving Intricacies of Language Processing,” in Proceedings of the
2022 2nd Asian Conference on Innovation in Technology (ASIANCON),
2022, pp. 1-7.

258
Authorized licensed use limited to: JNT University Kakinada. Downloaded on December 16,2024 at 10:50:11 UTC from IEEE Xplore. Restrictions apply.