odoo/enterprise#99500

Created by Fun Services, Prakash Prajapati (ppr)

label

odoo-dev:master-field-tracking-poc-ppr

head

67bb2ebdb01063e7d7b24694995336e4fd18f5a5

merged

2 months ago by Marketing, Thibault Delavallee (tde)

[REF] mail: add tracking values in message body

RATIONALE

Independent study [1] recently showed that storing tracking values takes
quite some db space for few "real usage". Storing tracking values directly
in body html is therefore considered as an alternative to using a table with
fields holding field ID, information, old and new values in a structured
and easy-to-manipulate way.

According to the study author [2], advantages are
* up to 30% db space saved;
* faster read when chatter is loaded as we can remove queries required to
access tracking value table, as well as field-based group check that
ensure security at field level;
* faster write (which globally never happen in real life use cases);
* globally faster update due to not having index on tracking table to
update;
* easier search, allowing to search words in body instead of having to
search in body + in field_id / old_value / new_value of tracking
value model;
* less custom code for audit trail, which does simple html formatting
based on tracking values;
* less custom code for formatting. Having directly html allows to remove
code that fetches tracking values (protected model), formats it and send
it to frontend JS chatter, which in turn has to define a model and
formatting for those;

Disadvantages that should not be considered as disadvantages are
* no more field-based security. It is known since tracking introduction
that field with group should not be tracked [3];
* loss of translation support: search on translated field names for
example is not possible, as field name is stored translated. However
this is not considered as an issue, because you generally search on
values (content) which is often language independent. Translated values
search was not correctly supported with tracking values (e.g. selection
fields), and this is not going to be improved with html. We could have
improved tracking value translation support but it was not considered
as being worth it;
* potential loss of protection against update. Previously tracking values
were not accessible to anyone (except admins), unless going through
dedicated methods or controllers. Now being able to update body means
being able to update tracking values inside it. Protection can be added
but checking pre- and post- content of html fields is notoriously fragile
and more easily bypassed. This was not considered as a real issue even
for audit trail;

Although original study author explains that tracking value were never used
in Odoo, it is also worth pointing out that removing tracking value model
leads to several changes in various addons:
* duration mixin rewrite. This mixin computes time spend in stages for
heavy production models like crm.lead, project.task, helpdesk.ticket,
hr.applicant. It was previously computed based on tracking values, aka
no extra db usage. We now have to store this time information as
retrieving it in html is not really scalable. This means a new ever
growing fields on those models, storing time information in json.
* project burndownchart rewrite. It was also based on tracking values
and therefore needs a complete rewrite.
* mrp.production requires a new stored field to find the last start
production date, previously fetched (without performance issue
being reported) in tracking values;
* hr.leave.allocation requires a new stored field to store allocation
change for accrual, previously fetched (without performance issue
being reported) in tracking values;
* hr.recruitment.report rewrite, as it was based on tracking values and
now uses stored duration mixin values;
* tracking values were also used for sale.order log and tracking but
"time issues" lead to rewrite them as a custom model instead of
tracking values a few years ago (without clear insights on what
was really the issue, original author of this change being the same
as the one requesting removal of tracking values);

Other authors [4] tend to disagree with the study author, pointing out that
* people actually track fields with groups -> this may lead to security
issues, or having to stop tracking those fields. Study author discards
this usage, considered as advanced / too niche;
* having to store duration information, allocation details, mrp production
date, ... is going to negatively impacts db storage gains. Study author
considers those are negligible without providing any numerical
value;
* storing html means no easy chatter reformat, as well as useless noise
stored in db (html nodes, class names, ...). Study author points out
that his study shows at least 30% gain whatever the situation;
* easier leak of tracked values: each person having access to the
message has access to body, hence tracking values. Previously tracking
values were never accessible (except admins), only given through
dedicated controllers or methods (e.g. discuss store);
* tracking values are used in production environment notably when
analyzing issues (e.g. previous state of records, ...), having a stored
structured way of finding information is valuable in this case. Doing
same analysis in html is going to be harder, especially with non
translated field names, parsing html, ...
* gained db space is anyway negligible compared to a full production
db (e.g. optimistic estimated 50 Gb for a 3 Tb on odoo.com). This is
anyway going to be erased by ever-growing size of db (accumulated
messages, activities, ...);

All this taken into account it was decided by R&D head to move to html
usage for tracking values, given the numerous advantages it gives. However
in order to still allow advanced usage of tracking values, a now optional
module holds the mail.tracking.value table and records creation when
performing tracking. With that module feature should globally be equivalent
(minus the field-level groups security support) for advanced / technical
usage.

DETAILS

Please also refer to community commits.

Commit 1: helpdesk: store stage durations according to company working hours

This commit updates helpdesk ticket stage durations based on the company’s
working hours. We now store the total working hours in duration_tracking,
which is used for ticket sla frozen hours instead of the tracking model.

Example: a ticket stays in a stage for 2 days. Previously, it recorded 24
hours, but now it stores 16 hours based on working hours.

This duration is used in sla policies, so there is no longer a need to
manually exclude non-working hours.

Commit 2: hr_recruitment_reports: refactor recruitment stage analysis report

Use stage duration to determine how long an applicant stays in each stage
which was previously based on the tracking model.

hr_recruitment_stage_report: convert the report to use the duration_tracking
format. start and end dates are now computed based on the time spent in each
stage.

Commit 3: account_followup, hr_payroll, helpdesk_stock: clean tracking & groups on fields

• Remove groups from fields to avoid restricting tracked values
• Add missing tracking=None where needed

Commit 4: l10n_us_hr_payroll: store allocation data changes

Previously to this commit accrual is computed based on tracking values linked
to 'number_of_days' field previously to the requested date. As we are
migrating away from stored tracking values we now store dedicated
information on allocation itself.

New field is a JSON holding date-based changes. Overall computation and
behavior should stay the same as before.

REFERENCES

Also see odoo/odoo#248505 (and linked PRs) for preliminary work cleaning and
improving tracking mechanism.

task-5260614

[1] cannot disclose it but it might come from a bigram who wants to remove
that table since (at least) 2017 and prefers to have each addon doing
some kind of custom tracking instead of a transversal approach;
[2] see 1;
[3] :shrug:
[4] globally most of R&D senior developers as depicted in January and February
internal guru meetings;

Co-Authored-By: Prakash Prajapati <ppr@odoo.com>
Co-Authored-By: Jigar Vaghela <jva@odoo.com>
Co-Authored-By: Fabien Pinckaers <fp@odoo.com>
Co-Authored-By: Thibault Delavallee <tde@odoo.com>