for class, i'm processing raw text documents (our examples include novels can downloaded gutenberg project) , turning them dictionary data structure. each word, need know paragraph(s) occurs in, , how many times in each paragraph.
the procedure can divided as
- break document words, removing whitespace, commas, periods, etc.
- for each document, iterate on words. in dictionary. if word exists, update entry. if doesn't exist, create new entry.
i'm doing matlab because rest of work in matlab , didn't want have deal language/environment. turns out matlab has pretty string processing functions. however, i'm disturbed how slow code running. first part mentioned above not problem; use parfor loop , goes pretty fast. second part problem is.
my initial attempt @ creating dictionary use structs, matlab built-in data type. idea create struct called dictionary, field names actual words, e.g., dictionary.the or dictionary.gnome words 'the' , 'gnome.' worked provided word valid field name (it couldn't contraction, instance). ran slow, sought different solution. next attempt use containers.map, matlab's equivalent of hash map or dictionary object. (one of coworkers informed me matlab inefficient @ looking field names of structs, whereas hash table has o(1) lookup time.) however, making substitution slowed down performance!
at point, feel i've pretty substantial attempts @ optimizing code , starting wonder if matlab sensible choice here. i'm trying figure out whether slowness product of matlab or of me being terrible programmer (normally matlab quite fast when comes linear algebra, arrays, , matrices). rather have read on code, i'd appreciate whatever feedback community has offer on either of following:
does type of thing (i.e., language processing) in matlab or similar language, such python? if so, i'd try benchmark of timings. (i've got macbook, 2.8 ghz processor, , i'm getting 10-20k words per second.)
is possible i'd better results if switched compiled language java or c++? ballpark estimate, kind of improvement?
my idiosyncratic opinion:
if project can nicely split between (1) text, data processing , (2) mathematical analysis, i'd of former in java or python. (note: i'm not nlp guy.)
matlab either amazingly fast (eg. matrix operations using blas/lapack libraries) or shocking slow depending on exactly doing.
there's no high performance hashmap solution in matlab. compared can in c++ or java, using containers.map not fast. (i've heard people advocating using struct, fieldname hack trick faster map performance in matlab.)
quick , dirty, horribly imprecise test:
inserting 1 million randomly generated, unique strings containers.map on machine takes me 30 seconds:
tic(); i=1:1000000 m(x{i}) = i; end toc(); elapsed time 38.781537 seconds. on other hand, invoking java virtual machine , running code below takes machine less 2 seconds:
hashmap<string, double> hm = new hashmap<string, double>(); random r = new random(); for(long = 0; < 1000000; i++) { hm.put(double.tostring(r.nextdouble()), r.nextdouble()); } mgunn@odysseus:~/matlab/delme/container_test$ time java ctest 1.697u 0.151s 0:00.91 202.1% 0+0k 0+1io 0pf+0w accessing java objects matlab has lot of overhead well...
tic(); h = java.util.hashmap; i=1:1000000 h.put(x{i},i); end toc(); elapsed time 53.177989 seconds. additional comments
for maximum performance, it's hard beat highly optimized c/c++ code. unless you're extremely familiar c/c++ though (and then), you'll end having headaches memory management etc....
for academic coding, excellent vs. ok performance isn't issue. getting code correct more important speed.
java or python easier languages code in, java tending have better performance of 2 (but i'm not expert here).
Comments
Post a Comment