mwetoolkit Code

* Word._extra was removed in favor of Word._props, which
stores every prop (even "surface", "lemma", "pos" and "syn").
Python `property` objects were added to allow for code like
`w.surface = w.lemma` to continue working.

* Word.__init__ does not accept **kwargs arguments anymore,
but requires the initial `props` dict.

* The weird method Word.__len__ was renamed to Word.wc_length.

* Ngram.set_all was only used to replace entries by WILDCARD.
Since we will soon delete WILDCARD, Ngram.set_all was removed
and Ngram.foreach_del_prop was created in its place.

* We now treat most files as unable to produce empty props.
For example, `going||V|` in Moses will have no lemma or syn,
while we previously generated lemma="" and syn="".
In the future, we will issue warnings for files that CAN and
DO have empty word props.

* fmtutil.make_props was created as a temporary solution to the fact that
XWEFormatter generates tuples instead of `props` dicts. In the future,
this should be fixed.

* Similar to fmtutil.make_props, we have now a WILDCARD checker
inside Word.__init__. In the future, this should disappear, as
everyone should use fmtutil or something like it.

* File `candidates.py` had special code for still using
"surface if lemma is unavailable". Now that WILDCARDs are disappearing,
a bug has been fixed and we ARE actually generating surface outputs
if lemma is unavailable. Is this really the desired outcome?