少样本NER研究：基于Transformer和预训练语言模型的综合探索
下载需积分: 5 | 733KB |
更新于2024-08-05
| 5 浏览量 | 4 评论 | 举报
1
收藏
立即下载 "微软与UIUC韩家炜团队合作发表了一篇关于少样本NER(命名实体识别)的全面研究论文，探讨在有限标注数据条件下如何有效地构建NER系统。该研究基于最新的基于Transformer的自监督预训练语言模型(PLMs)，并提出了三种正交策略来提升模型在少样本情况下的泛化能力：(1)使用元学习构建不同实体类型的原型，(2)通过有噪声的网络数据进行监督预训练以提取与实体相关的通用表示，(3)利用未标注的领域内数据进行自我训练。文章还研究了这些策略的不同组合，并在10个公共NER数据集上进行了广泛的实证比较，涵盖了不同比例的标注数据。"
在这篇论文中，研究者关注的核心问题是如何在标记数据稀缺的环境中优化NER系统的性能。传统的机器学习和深度学习模型通常依赖大量标记数据来训练，但在少样本情况下，这些模型可能会过拟合训练数据，导致泛化能力下降。为了解决这个问题，论文提出了以下三个关键策略：
1. **元学习构造原型**：元学习是一种让模型能够快速适应新任务的学习策略。在NER的背景下，这可能意味着通过元学习方法创建代表各种实体类型的原型，使模型能够在有限的样本中学习到更通用的特征，从而提高对新实体类型的识别能力。
2. **监督预训练**：研究人员使用含有噪声的网络数据进行预训练，以获取与实体相关的通用表示。这种方法可以增强模型对实体的理解，即使这些数据没有经过精细的标注。预训练后，模型将具备一定的处理未见过的实体类型的能力。
3. **自我训练**：自我训练是一种无监督学习策略，利用未标注的数据来改进模型。在这种情况下，模型先基于少量的标注数据进行初步训练，然后用预测的标签去指导对未标注数据的进一步学习，形成迭代过程，逐渐提升模型的性能。
通过组合这三个策略，研究者进行了广泛实验，评估了它们在不同比例的标注数据上的表现。这样的工作对于实际应用非常有价值，因为在现实世界中，获取大量精确标注的数据往往是昂贵且耗时的。这篇论文为解决这个问题提供了一个系统性的框架和深入的洞察，为未来的研究提供了新的方向。
Mr. Bush
asked
Congress
to raise to
$ 6 billion
Org
Money
Person Others
Transformer-based Backbone Network
Linear Layer + SoftMax
Org M M MO O OOP P
Model:
Target:
Input Sentence:
Entity Types:
Mr. Bush
asked
Congress
to raise to
$ 6 billion
Person
Org Money
Support set:
Gates
co-founded Microsoft …Query set:
Jobs
founded
NeXT Inc
. with
$ 7 million
Distance
SoftMax
(a) Baseline: NER with a linear classifier (b) Prototype-based method
Series 5
runners up
JLS
and
Florence and the Machine
performed on show
Event Musician Artist
Entity Types:
Input Sentence:
Examples
Others
Labeled set
Unlabeled set
Student
Teacher
O M OP
…
Distillation
(c) Noisy supervised pre-training (d) Self-training
Figure 2: Illustration of different methods for few-shot NER. In this example, each token in the input sentence is
categorized into one of the four entity types. (a) A typical NER system, where a linear classifier is built on top of
unsupervised pre-trained Transformer-based networks such as BERT/Roberta. (b) A prototype set is constructed
via averaging features of all tokens belonging to a given entity type in the support set (e.g., the prototype for
Person is an average of three tokens: Mr., Bush and Jobs). For a token in the query set, its distances from
different prototypes are computed, and the model is trained to maximize the likelihood to assign the query token
to its target prototype. (c) The Wikipedia dataset is employed for supervised pre-training, whose entity types are
related but different (e.g., Musician and Artist are more fine-grained types of Person in the downstream
task). The associated types on each token can be noisy. (d) Self-training: An NER system (teacher model) trained
on a small labeled dataset is used to predict soft labels for sentences in a large unlabeled dataset. The joint of the
predicted dataset and original dataset is used to train a student model.
How to leverage unlabeled in-domain sentences in
a semi-supervised manner? Note that these three
directions are complementary to each other, can be
further used jointly to extrapolate the methodology
space in Figure 1.
3.1 Prototype-based Methods
Meta-learning (Ravi and Larochelle, 2017) have
shown promising results for few-shot image classi-
fication (Tian et al., 2020) and sentence classifica-
tion (Yu et al., 2018; Geng et al., 2019). It is natural
to adapt this idea to few-shot NER. The core idea is
to use episodic classification paradigm to simulate
few-shot settings during model training. Specifi-
cally in each episode,
M
entity types (usually
M <
|Y|
) are randomly sampled from
D
L
, containing a
support set
S = {(X
i
, Y
i
}
M×K
i=1
(
K
sentences per
type) and a query set
Q = {(
ˆ
X
i
,
ˆ
Y
i
}
M×K
0
i=1
(
K
0
sentences per type).
We build our method based on prototypical net-
work (Snell et al., 2017), which introduces the no-
tion of prototypes, representing entity types as vec-
tors in the same representation space of individual
tokens. To construct the prototype for the
m
-th
entity type
c
m
, the average of representations is
computed for all tokens belonging to this type in
the support set S:
c
m
=
1
|S
m
|
X
x∈S
m
f
θ
0
(x), (3)
where
S
m
is the tokens set of the
m
-th type in
S
,
and
f
θ
0
is defined in
(2)
. For an input token
x ∈ Q
from the query set, its prediction distribution is
computed by a softmax function of the distance be-
tween
x
and all the entity prototypes. For example,
the prediction probability for the
m
-th prototype is:
q(y =I
m
|x)=
exp (−d(f
θ
0
(x), c
m
))
P
m
0
exp (−d(f
θ
0
(x), c
m
0
))
(4)
where
I
m
is the one-hot vector with 1 for
m
-th
coordinate and 0 elsewhere, and
d(f
θ
0
(x), c
m
) =
kf
θ
0
(x) − c
m
k
2
is used in our implementation.
We provide a simple example to illustrate the
prototype method in Figure 2(b). In each train-
ing iteration, a new episode is sampled, and the
model parameter
θ
0
is updated via plugging
(4)
into
(1)
. In testing phase, the label of a new token
x
is assigned using the nearest neighbor criterion
arg min
m
d(f
θ
0
(x), c
m
).
3.2 Noisy Supervised Pre-training
Generic representations via self-supervised
pre-trained language models (Devlin et al.,
剩余11页未读，继续阅读
相关推荐
资源评论
狼You
2025.07.13
韩家炜组的这一综述，对于理解少样本NER领域具有重要意义。🍖
艾斯·歪
2025.05.20
面对数据稀缺挑战，这篇文章为少样本NER的未来发展指明了方向。
韩金虎
2025.04.28
微软联合UIUC推出少样本NER综述，为标注数据稀缺问题提供解决方案。
袁大岛
2025.04.24
综述深入探讨少样本条件下的命名实体识别技术，内容丰富详实。
syp_net
- 粉丝: 157
我的内容管理
展开
最新资源
- 在Docker中搭建CI仪表环境的实用指南
- 在Docker上成功运行GUI应用程序的完整指南
- GitLab Sprint Metrics报告生成器：Java脚本解析与应用
- Delphi实现远程桌面控制与操作指南
- TryChain: 探索Java区块链技术的触感
- Java Spring Boot实现的设备发现服务及REST API使用指南
- React与Express构建的邮件注册页面：集成reCaptcha和API代理
- Dell服务器电源热监控微服务Java实现与部署指南
- Nagios被动代理的跨平台部署与维护指南
- 简化DDD实践指南：结合Clean Architecture实现微服务
- PS4游戏转储工具 ps4-dumper-vtx 使用教程
- 《SaltStack技术入门与实战》实践案例分析
- SLAE项目中Python实现HTTP分配机制的研究
- 自动化美国签证预约通知工具使用指南
- 利用LibGDX掌握2D游戏开发的Udacity课程资源
- 独立部署mecab服务器：Docker化Python应用教程
- grin-explorer: Grin区块链资源的Python管理工具
- 利用Pythia PRF协议，SafeID库加强Web服务器密码安全
- git-terminal: 实现多git仓库监控与管理的微型终端仪表板
- 一键部署Hackazon：使用Docker运行多服务环境
- Delphi网络聊天室源码解析与应用
- 掌握数学与代码，深入瓦尔拉斯均衡大师班
- Perl实现的LoveLive μ's偶像成员信息展示脚本
- 掌握Python实现复杂区块链系统的要点
