Use of digit groups separator requires presence of decimals

[ony]

Consider example:

% COLUMNS=80 hledger reg -f -
commodity EUR
	format 1,000.00 EUR

commodity UAH
	format 1.000,00 UAH

2017/1/1
	expenses  1,000 EUR
	expenses  1.000 EUR
	expenses  1,000.00 EUR
	assets

2017/1/1
	expenses  1,000 UAH
	expenses  1.000 UAH
	expenses  1.000,00 UAH
	assets

This results in

2017/01/01                      expenses                  1.00 EUR      1.00 EUR
                                expenses                  1.00 EUR      2.00 EUR
                                expenses              1,000.00 EUR  1,002.00 EUR
                                assets               -1,002.00 EUR             0
2017/01/01                      expenses                  1,00 UAH      1,00 UAH
                                expenses                  1,00 UAH      2,00 UAH
                                expenses              1.000,00 UAH  1.002,00 UAH
                                assets               -1.002,00 UAH             0

Which is pretty unexpected.
It should be:

2017/01/01                      expenses              1,000.00 EUR  1,000.00 EUR
                                expenses                  1.00 EUR  1,001.00 EUR
                                expenses              1,000.00 EUR  2,001.00 EUR
                                assets               -2,001.00 EUR             0
2017/01/01                      expenses                  1,00 UAH      1,00 UAH
                                expenses              1.000,00 UAH  1.001,00 UAH
                                expenses              1.000,00 UAH  2.001,00 UAH
                                assets               -2.001,00 UAH             0

I have to be very careful when I write numbers with thousands and always put decimals separator to avoid treating my numbers smaller than they are in fact.

hledger should honor decimals and digit groups specification of commodity format.

I don't mind to use space as a separator, but this isn't allowed yet. See #330 .

[simonmichael]

It's not hard at all to confuse the style inferrer. Does specifying the style with a commodity directive help ?

Given a clear specification of what should happen above, we could improve style inference, maybe. I'm not sure it's the right approach though.

[simonmichael]

Oh you gave the expected result. I admit I haven't spent the time to understand what's happening here. Given the difficulty of inferring style correctly for all possible confusing inputs, I'm not even sure yet if this is a bug, or something we can't reasonably handle. Improvements to code or docs welcome.

[ony]

@simonmichael, example already includes commodity directive. That's the problem. I believe inferred styles have no effect on this. I guess this happens during parse of the amount. It simply parse both 30.2 and 30,2 equally without using information about commodity format.

P.S. Re inferred styles. Inferred amounts always use inferred styles and ignore declared ones.

[ony]

At this moment I use hledger-check.hs to verify that number style is consistent (same precision and decimals separator).

[ony]

It is possible to solve it in an ugly way within current design. I.e. post-process entries without digit groups and that have decimal separator matching with commodity specification. But consider next journal:

commodity  $100,000.00
2016/1/1
    a  $-10.00
    b

commodity  $100.000,00
2017/1/1
    a  $-10,00
    b

I would expect in both cases it to be parsed as $-10 (i.e. last commodity directive have effect until next occurrence ).

[simonmichael]

Note to self: check fix also resolved #545.

[dashed]

This bit me when I tried hledger on files, originally procesed with ledger-cli v3, that have balances like 4,000 CAD (the comma is mis-interpreted as a decimal placeholder) :( And I have a lot of transactions of this form.

Is a fix being worked on?

I'd like to help, but I'm not proficient with haskell.

[simonmichael]

@dashed nobody is currently working on it; this issue is available, and so is haskell mentoring (in #hledger) :)

I think @ony's example above is more complicated than the norm. You can usually prevent this problem by simply ensuring the first amount of this commodity in the journal (or a commodity directive for this commodity) has both a digit group separator and a decimal point.

[ony]

@simonmichael , commodity only contributes to how we render amount, as I know. This error happens during parsing and we don't have commodity directive there.
To get commodity influence how we parse we should have two pass parser (delay amount parsing) or context-aware parser (to get effect on rest of the file).

[dashed]

maybe another directive?

[ony]

@dashed , I think we can re-use existing directive. It just a lot of changes associated with migrating from TextParser to JournalParser. You'll need to change all usages.
Besides that you'll need a way to to supply journal when amount from arguments being parsed (yes, that's another place this issue exposed).

Update: actually numberp have no information about symbol. So maybe we can just extend it.