odoo/odoo#103697

Created by Accounting, William André (wan)

label

odoo-dev:16.0-dashboard-perf-wan

head

34a34842417e1984422577b392b7aa8e5b4cfbcb

merged

3 years ago by Accounting, Quentin De Paoli (qdp)

	odoo/odoo	odoo/enterprise
16.0	#103697	#33165
saas-16.1	#117929	#39440
saas-16.2	#117949	#39457
17.0
18.0
saas-18.2
saas-18.3
saas-18.4
19.0
master	#117964	#39461

[IMP] account: dashboard performance

As the number of account.move, account.move.line,
account.bank.statement.line and account.journal is growing, the
accounting dashboard, which is the entry point of the app, gets slower
and slower.

There are multiple issues being addressed in this commit:
* The data for each journal is computed journal by journal. This means
that the number of queries run increases linearly with the number of
journals. While the boilerplate around running multiple queries is
negligible compared to the running time of the queries in this case,
some queries take as much time to run for one journal or for many.
To improve this, all the queries are now batched. This has been done
by refactoring the code; all these functions are now called on as many
records as needed¹:
- _get_journal_bank_account_balance
- _get_journal_outstanding_payments_account_balance
- _get_last_bank_statement
- get_line_graph_datas
- get_bar_graph_datas
- get_journal_dashboard_datas
* The gap detection and the entries' count are computed fields
(has_sequence_holes and entries_count respectively). We don't need
to display/compute these fields for all types of journals, but since
they were mentioned by using a <field/> node in the view, they were
computed for all journals displayed. Instead of using the <field/>
node, we are now setting the value in the kanban_dashboard field.
* Documents in foreign currencies on journals in foreign currencies need
to get the rate in order to be aggregated in the journal's currency.
There are 3 cases:
- Document in journal's currency
- Company's currency is the same as the journal's
- Document, company and journal have 3 different currencies
Before this commit, the second case will still fetch the daily rate
for the document in order to do the conversion, but we actually
already know the conversion; it is stored on the document.

Benchmark

On a populate database with:
- 4 res.company (with accounting enabled)
- 45 account.journal
- 19k account.move
- 140k account.move.line
- 4k account.bank.statement.line
- 4 account.bank.statement

	Query count	Query time	Remaining time
Before fix	279	0.333s	0.375s
After fix (without update)	40	0.120s	0.170s
After fix (with update2)	38	0.100s	0.170s

Note that the currency conversion was disabled because the populate
database doesn't represent a realistic dataset regarding this. Disabling
it only improves the numbers before the fix.

Note

A lot of the time remaining comes from the aggregation of
draft/unpaid invoices with the correct rate done in python instead of in
SQL. This commit doesn't change the behavior but this could be rethought
from a function point of view.