WikiPedia 全文のダウンロード


wiki-latest-pages-articles.xml.bz2

python jawikicorpus.py /path/to/jawiki-latest-pages-articles.xml.bz2 jawiki

https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/scripts/make_wikicorpus.py


python jawikicorpus.py /path/to/jawiki-latest-pages-articles.xml.bz2 jawiki

MmCorpus.serialize(dst + '_bow.mm', wiki, progress_cnt=10000, metadata=True)

使ってみる
MmCorpus.serialize(dst + '_bow.mm', wiki, progress_cnt=10000, metadata=True)