Verification & Limitations

Verification:

Initially, the Balancer Simulations pool model was verified with python unit tests, getting data from local ganache EVM, and running tests using the balancer-core repo from Balancer. In order to keep as much precision as possible (Balancer operates with 18 decimals, in wei), we used Python's Decimal Type and normalized all token amounts to Ether unit, operating with actual decimals, unlike wei in Solidity. Different decimal values for ERC20 tokens are taken into account in the pulldata.py parsing script, for example USDC has 6 decimals, while most ERC-20 tokens have 18.

Generating single tests running BPool contracts with Ganache was slow, fragile and difficult for simulating large numbers of transactions. Hence, we improved our parsing script:

1. Get and process data from Ethereum ETL in BigQuery, including non-anonymous events emitted by Balancer contracts. With every output of a transaction against a Balancer Pool being available in the data set, we can easily infer swap, joins and exit outcomes.

2. Get all transactions to decode the user input and the actual Balancer method applied.

In order to get pool state and user inputs to code policy tests, we ran the simulation up to the desired action, recorded the previous pool state, used contract_call data as inputs (or action data in a simplified mode), run our model and compared with action output (actual outcomes).

Test example:

def test_p_join_pool_simplified_contract_call(self):
    initial_weth_balance = Decimal('67754.45880861386396117576576')
    initial_dai_balance = Decimal('10016378.43379686305875979834')
    initial_pool_shares = Decimal('100.0035090123482194137033160')
    pool = {
        'tokens': {
            'WETH': Token(bound=True, weight=Decimal('0.1'), denorm_weight=Decimal('40'), balance=initial_weth_balance),
            'DAI': Token(bound=True, weight=Decimal('0.4'), denorm_weight=Decimal('10'), balance=initial_dai_balance),
        },
        'generated_fees': {
            'WETH': Decimal('0'),
            'DAI': Decimal('0')
        },
        'pool_shares': initial_pool_shares,
        'swap_fee': Decimal('0.0025')
    }
    current_state = {
        'pool': pool
    }
    action = {'pool_amount_out': '0.000029508254125206', 'type': 'join',
              'tokens_in': [{'amount': '0.019993601301505542', 'symbol': 'WETH'}, {'amount': '2.954876765664920082', 'symbol': 'DAI'}]}
    contract_call = {'type': 'joinPool', 'inputs': {'poolAmountOut': '0.000029508254125206', 'maxAmountsIn': None}}

    input_params, output_params = PoolMethodParamsDecoder.join_pool_contract_call(action, contract_call)
    answer = p_join_pool(params={}, step=1, history={}, current_state=current_state, input_params=input_params,
                         output_params=output_params)
    self.assertAlmostEqual(answer['tokens']['WETH'].balance, initial_weth_balance + Decimal('0.019993601301505542'), 5)
    self.assertAlmostEqual(answer['tokens']['DAI'].balance, initial_dai_balance + Decimal('2.954876765664920082'), 2)
    self.assertAlmostEqual(answer['pool_shares'], initial_pool_shares + Decimal('0.000029508254125206'), 5)

Conclusions:

There is a small error introduced. Most methods pass the test with an accuracy of at least 7 decimal places, except:

• join (5 – 2 decimal place precision)

• exit swap (4 decimal place precision in token amount)

• exit ( 2 – 4 decimal place precision in token amount)

Overall, the ratio calculation and division used for join and exit methods is less precise that in BMath swaps, even when using a Decimal type with 28 decimal place precision.

For additional confidence in the information gained from the model, there's a dedicated notebook to compare on-chain pool transactions with the simulation: NB #0: Sanity Check NB1 - 0x8b6-V1.0

This Sanity Check offers parameter sweeps to run and compare Balancer Simulations in three different modes of operation:

1. SIMPLIFIED: Update pool balances via state updates (simulation), use BigQuery event data as input, and apply most popular contract methods (such as using 'out-iven-in' for swaps only, since we can't know what the user chose, see below=

2. CONTRACT_CALL: Update pool balances via state updates (simulation),

   use the actual pool method and inputs decoded from transaction data

3. REPLAY_OUTPUT: Update pool balances (no simulation), display the BigQuery events 1:1 for reference.

Limitations

a) In-Given-Out not applied in simplified mode simulations Balancer Simulations V1.0 provides a parsing script to detect on-chain transactions and map it to state update functions. For swaps, we're using just one type of transactions 'out-given-in', and exclude 'in-given-out'.

pool_state_updates.py

b) Network-specific aspects

Being a Python digital twin of the system, Balancer Simulations can't reproduce some aspects of the orginal EVM network, such as transaction processing time, network pollution, or gas fees. As in any system simulation, the impact of such parameters needs to be carefully weighted, and there might be cases requiring a different simulation environment.