色とか、 - 271108の記憶貯蔵庫

~~言葉ってめんどくさい。色とか、気持ちで伝えればいいのに。~~

f:id:Amesiorine271108:20210413192952p:plain — 考えたこと

この時点で思いつく問題点

データベースどうやってつくるんだっけ？
未知の単語とかありそう。

とりあえず前回のメモを見つつ単語、読み、品詞にバラすものを作

from sudachipy import tokenizer
from sudachipy import dictionary
tokenizer_obj = dictionary.Dictionary().create()

def tokenize(x):
    sen = []
    voidwordlist = ("補助記号","記号","空白")
    dic = {"w":"<BOS>","pron":"","PoSp":["BOS/EOS", "*", '*', '*', '*', '*', "BOS/EOS"]}
    sen.append(dic)
    for y in tokenizer_obj.tokenize(x, tokenizer.Tokenizer.SplitMode.C):
        dic = {}
        dic["w"] = y.surface()
        if y.part_of_speech()[0] in voidwordlist:
            dic["pron"] = ""
        else:
            dic["pron"] = y.reading_form()
            
        dic["PoSp"] = y.part_of_speech()
        sen.append(dic)
    dic = {"w":"<EOS>","pron":"","PoSp":["BOS/EOS", "*", '*', '*', '*', '*', "BOS/EOS"]}
    sen.append(dic)
    
    return sen