jieba自定义词库分词并进行tfidf计算

最新推荐文章于 2024-10-26 17:12:06 发布

原创

最新推荐文章于 2024-10-26 17:12:06 发布 · 1.1k 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#python

1、导入库

import jieba
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer

2、将df_all_info的’JOB_TITLE’字段的内容都作为自定义词库，命名为job_title.txt

with open('D:\\proj\\sodic_2021\\job_title.txt','w',encoding='utf-8') as f:
    for i in range(len(df_all_info)):
        job_title=df_all_info.loc[i,'JOB_TITLE']
        f.write('\n%s' %(job_title))

3、将df_all_info中每个样本的’SPECILTY’字段进行分词，保存在’SPECILTY_JIEBA’字段中

df_all_info['SPECILTY_JIEBA']=0
jieba.load_userdict('D:\\proj\\sodic_2021\\job_title.txt')

for i in range(len(df_all_info)):
    if df_all_info.loc[i,'SPECILTY'] is np.NAN:
        continue
    else:
        word=jieba.cut(df_all_info.loc[i,'SPECILTY'])
        word_cut=''
        for j in word:
            word_cut=word_cut+' '+j.up