python实现中文分词FMM算法实例
本文实例讲述了python实现中文分词FMM算法。分享给大家供大家参考。具体分析如下:
FMM算法的最简单思想是使用贪心算法向前找n个,如果这n个组成的词在词典中出现,就ok,如果没有出现,那么找n-1个...然后继续下去。假如n个词在词典中出现,那么从n+1位置继续找下去,直到句子结束。
importre defPreProcess(sentence,edcode="utf-8"): sentence=sentence.decode(edcode) sentence=re.sub(u"[。,,!……!《》<>\"'::?\?、\|“”‘';]","",sentence) returnsentence defFMM(sentence,diction,result=[],maxwordLength=4,edcode="utf-8"): i=0 sentence=PreProcess(sentence,edcode) length=len(sentence) whilei<length: #findtheasciiword tempi=i tok=sentence[i:i+1] whilere.search("[0-9A-Za-z\-\+#@_\.]{1}",tok)<>None: i=i+1 tok=sentence[i:i+1] ifi-tempi>0: result.append(sentence[tempi:i].lower().encode(edcode)) #findchineseword left=len(sentence[i:]) ifleft==1: """goto4stepovertheFMM""" """shouldweaddthelastone?Yes,ifnotblank""" ifsentence[i:]<>"": result.append(sentence[i:].encode(edcode)) returnresult m=min(left,maxwordLength) forjinxrange(m,0,-1): leftword=sentence[i:j+i].encode(edcode) #printleftword.decode(edcode) ifLookUp(leftword,diction): #findtheleftwordindictionary #it'stherightone i=j+i result.append(leftword) break elifj==1: """onlyoneword,addintoresult,ifnotblank""" ifleftword.decode(edcode)<>"": result.append(leftword) i=i+1 else: continue returnresult defLookUp(word,dictionary): ifdictionary.has_key(word): returnTrue returnFalse defConvertGBKtoUTF(sentence): returnsentence.decode('gbk').encode('utf-8') dictions={} dictions["ab"]=1 dictions["cd"]=2 dictions["abc"]=1 dictions["ss"]=1 dictions[ConvertGBKtoUTF("好的")]=1 dictions[ConvertGBKtoUTF("真的")]=1 sentence="asdfa好的是这样吗vasdiw呀真的dafdasfiwasid是吗?" s=FMM(ConvertGBKtoUTF(sentence),dictions) foriins: printi.decode("utf-8") test=open("test.txt","r") forlineintest: s=FMM(CovertGBKtoUTF(line),dictions) foriins: printi.decode("utf-8")
运行结果如下:
asdfa
好的
是
这
样
吗
vasdiw
呀
真的
daf
dasfiw
asid
是
吗
?
希望本文所述对大家的Python程序设计有所帮助。