From 9b2cd724b52e32b22c11bbd496b7257ba651f98a Mon Sep 17 00:00:00 2001 From: Sylvain LE GAL Date: Tue, 5 Nov 2024 17:25:10 +0100 Subject: [PATCH] wip --- .markdownlint.rb | 4 + .pre-commit-config.yaml | 8 + CONTRIBUTORS.md | 14 +- DEVELOP.md | 110 +++++---- INTERNAL_NOTES.md | 69 ------ README.bk | 449 ++++++++++++++++++++++++++++++++++ README.md | 520 ++++------------------------------------ ROADMAP.md | 45 ---- 8 files changed, 572 insertions(+), 647 deletions(-) create mode 100644 .markdownlint.rb delete mode 100644 INTERNAL_NOTES.md create mode 100644 README.bk delete mode 100644 ROADMAP.md diff --git a/.markdownlint.rb b/.markdownlint.rb new file mode 100644 index 0000000..a190c0c --- /dev/null +++ b/.markdownlint.rb @@ -0,0 +1,4 @@ +all +# MD033 Inline HTML +# TODO: Tweak elements when https://github.com/markdownlint/markdownlint/issues/118 will be done? +exclude_rule 'MD033' \ No newline at end of file diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 7cce82a..4250433 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -30,6 +30,14 @@ repos: rev: "6.1.0" hooks: - id: flake8 + + - repo: https://github.com/igorshubovych/markdownlint-cli + rev: "v0.41.0" + hooks: + - id: markdownlint + args: [ --disable=MD033 ] + + # - repo: https://gitlab.com/smop/pre-commit-hooks # rev: v1.0.0 # hooks: diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 903091e..b9ec520 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -1,4 +1,6 @@ -# Developers +# Contributors + +## Developers * Sylvain LE GAL, from [GRAP](http://www.grap.coop), since March 2022 * Rémy TAYMANS, from [Coop It Easy](https://coopiteasy.be/), since June 2022 @@ -9,9 +11,11 @@ * Hugues DE KEYSER, from [Coop It Easy](https://coopiteasy.be/), since April 2024 * Gabriel PICKENHAYN, since May 2024 * Boris GALLET, since September 2024 -* Ahmet YIĞIT BUDAK, from [Altinkaya](https://www.altinkaya.com/fr), since October 2024 -* Alexandre AUBIN, from [Coopaname](https://www.coopaname.coop/), since October 2024 +* Ahmet YIĞIT BUDAK, from [Altinkaya](https://www.altinkaya.com/fr), + since October 2024 +* Alexandre AUBIN, from [Coopaname](https://www.coopaname.coop/), + since October 2024 -# Reviewers +## Reviewers -* Sébastien BEAU, from Akretion (https://akretion.com) +* Sébastien BEAU, from [Akretion](https://akretion.com) diff --git a/DEVELOP.md b/DEVELOP.md index 866e315..f16f19f 100644 --- a/DEVELOP.md +++ b/DEVELOP.md @@ -1,8 +1,10 @@ -# Installation to develop +# Develop -## Basic installation +## Installation to develop -``` +### Basic installation + +```shell git clone https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/ cd odoo-openupgrade-wizard virtualenv env --python=python3 @@ -12,19 +14,20 @@ poetry install ``odoo-openupgrade-wizard`` commands are now available in your virutalenv. -## Advanced installation +### Advanced installation If you want to use this library without installing anything in your system, execute the following steps, otherwise, go to 'Installation' part. -1. Run a docker container: - -``docker run -it ubuntu:focal`` - -2. Execute the following commnands +- Run a docker container: +```shell +docker run -it ubuntu:focal ``` +- Execute the following commnands + +```shell apt-get update apt-get install git python3 python3-pip python3-venv @@ -37,40 +40,40 @@ pipx install virtualenv pipx install poetry ``` -# Run tests +## Run tests -## Via pytest (simple) +### Via pytest (simple) This will run tests only for the current ``python3.X`` version. (in your virtualenv) -``` + +```shell poetry run pytest --cov odoo_openupgrade_wizard --verbosity=2 --exitfirst ``` -## Via Tox (advanced) +### Via Tox (advanced) This will run tests for all the python versions put in the ``tox.ini`` folder. (in your virtualenv) -``` + +```shell tox ``` Note : you should have all the python versions available in your local system. - -``` +```shell sudo apt-get install python3.6 python3.6-distutils sudo apt-get install python3.7 python3.7-distutils sudo apt-get install python3.8 python3.8-distutils sudo apt-get install python3.9 python3.9-distutils ``` -## Via Gitlab Runner locally +### Via Gitlab Runner locally - -``` +```shell # Install tools pipx install gitlabci-local @@ -78,17 +81,19 @@ pipx install gitlabci-local gitlabci-local ``` -# Debugging +## Debugging Some docker command could help you when using / developping this tools. -**Enter the postgres container** +To Enter the postgres container, run: +```shell docker exec -it POSTGRES_CONTAINER_NAME /bin/bash +``` -# Contribute +## Contribute -## Provide newsfragments in your merge requests +### Provide newsfragments in your merge requests If you propose a merge request, please add a newsfragments with it. @@ -115,11 +120,12 @@ a `+` symbol. This is a example of newsfragments. `newsfragments/+sub-command-cowsay.feature`: + ``` Adds a new subcommand `cowsay` to allow poeple to speak like a cow. ``` -## Add python dependencies +### Add python dependencies If you add new dependencies, you have to: @@ -127,7 +133,7 @@ If you add new dependencies, you have to: - run the following command in your virtualenv : ``poetry update`` -## Release on Gitlab and publish on PyPI +### Release on Gitlab and publish on PyPI Building, releasing and publishing a new version works with tags. @@ -143,7 +149,7 @@ on gitlab, but they trigger a publication on PyPI. Before creating a tag, ensure that the version of the program is updated. To update the program version you can use the command: -``` +```shell poetry version {major,minor,patch} ``` @@ -163,27 +169,26 @@ To see if the publication on PyPI and the release on gitlab were done correctly, go in Build > Pipelines. You will find a pipeline for the tag you just created. +## Understanding the library -# Understanding the library - -## Tools to understand +### Tools to understand The library is using many tools. It is recommanded to understand that tools to contribute to that project: -* Docker (https://www.docker.com/) -* Gitlab CI (https://docs.gitlab.com/ee/ci/quick_start/index.html) -* openupgrade project (https://github.com/oca/openupgrade) and related openupgradelib (https://github.com/oca/openupgradelib) -* poetry (https://python-poetry.org/) -* odoorpc (https://github.com/OCA/odoorpc) -* git-aggregator (https://github.com/acsone/git-aggregator) +- Docker () +- Gitlab CI () +- openupgrade project () + and related openupgradelib () +- poetry () +- odoorpc () +- git-aggregator () Also this project is inspired by the following tools: -* click-odoo-contrib (https://github.com/acsone/click-odoo-contrib) +- click-odoo-contrib () - -# Dockerfile information +## Dockerfile information ### From version 5 to 7 @@ -195,61 +200,68 @@ Try to create dockerfile, based on the odoo official ones fails. Any help welcom ### From version 11.0 to latest version. (Python3) -The Dockerfile of the version 11 to the lastest version of Odoo are written this way : +The Dockerfile of the version 11 to the lastest version of Odoo are written +this way: -- Copy the content of https://github.com/odoo/odoo/blob/ODOO_VERSION/setup/package.dfsrc +- Copy the content of - remove all the part after the big ``apt-get install`` -- install debian package ``git`` to have the possibility to pip install from git url. +- install debian package ``git`` to have the possibility + to pip install from git url. - install custom debian packages - install python odoo requirements -- install python ``setuptools-scm`` lib to have the possibility to pip install ``openupgradelib`` from git url. +- install python ``setuptools-scm`` lib to have the possibility + to pip install ``openupgradelib`` from git url. - install python custom requirements - makes link between external user and docker odoo user -## Réferences +## References - how to install gitlab runner locally: -https://docs.gitlab.com/runner/install/linux-manually.html + - Check your CI locally. (French) -https://blog.stephane-robert.info/post/gitlab-valider-ci-yml/ - -https://blog.callr.tech/building-docker-images-with-gitlab-ci-best-practices/ + + ## Python version settings depending on the debian version This part can help you if you want to change your autogenerated Dockerfiles. -See (https://github.com/odoo/odoo/blob/ODOO_VERSION/setup/package.dfdebian) +See (). ### debian:wheezy (V7) (for Odoo: 8.0) + - Ubuntu release : 12.04 xxx, 12.10 xxx, 13.04 xxx, 14.10 xxx - python2.7 - First release : 04/05/2013 - End LTS : May 2018 ### debian:jessie (V8) (for Odoo: 9.0, 10.0) + - Ubuntu release : 14.04 trusty, 14.10 utopic, 15.04 vivid, 15.10 wily - python2.7 - First release : 26/04/2015 - End LTS : June 2020 ### debian:stretch (V9) (for Odoo: 11.0, 12.0) + - Ubuntu releases : 16.04 xenial, 16.10 yakkety, 17.04 zesty, 17.10 artful - python2.7 and python3.5 - First release : 17/06/2017 - End LTS : June 2022 ### debian:buster (13.0, 14.0) + - Ubuntu release : 18.04 bionic, 18.10 cosmic, 19.04 disco, 19.10 eoan - python2.7 and python3.7 - First release : 06/07/2019 - End LTS : Undefined. -## debian:bullseye (15.0, 16.0) +### debian:bullseye (15.0, 16.0) + - Ubuntu release : 20.04 focal, 20.10 groovy, 21.04 hirsute, 21.10 impish - python3.9 - First release : 14/07/2021 diff --git a/INTERNAL_NOTES.md b/INTERNAL_NOTES.md deleted file mode 100644 index 20dcffc..0000000 --- a/INTERNAL_NOTES.md +++ /dev/null @@ -1,69 +0,0 @@ -# TODO - -* with coop it easy : -- short_help of group decorator ? seems useless... - -* add constrains on ``--step`` option. - -* revert : set 777 to log and filestore to be able to write on this folder - inside the containers. TODO, ask to coop it easy or commown for better alternative. - -* allow to call odoo-bin shell, via : https://github.com/d11wtq/dockerpty - (see https://github.com/docker/docker-py/issues/247) - - -# List of the series of odoo -# python version is defined, based on the OCA CI. -# https://github.com/OCA/oca-addons-repo-template/blob/master/src/.github/workflows/%7B%25%20if%20ci%20%3D%3D%20'GitHub'%20%25%7Dtest.yml%7B%25%20endif%20%25%7D.jinja - - -# tips -``` -# execute sql request in postgres docker -docker exec db psql --username=odoo --dbname=test_v12 -c "update res_partner set ""email"" = 'bib@bqsdfqsdf.txt';" -``` - -# TODO Nice To have - -- Fix gitlabci-local. For the time being, it is not possible to debug - locally. (there are extra bugs locally that doesn't occures on gitlab, - in ``cli_B_03_run_test.py``... - - -- add - -# Try gitlab runner - -curl -LJO "https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_amd64.deb" - -sudo dpkg -i gitlab-runner_amd64.deb - -(https://docs.gitlab.com/runner/install/linux-manually.html) - - -# TODO: -- check dynamic user id with -https://github.com/camptocamp/docker-odoo-project/blob/master/bin/docker-entrypoint.sh - - -in modules.csv.j2 : -# TODO, this value are usefull for test for analyse between 13 and 14. -# move that values in data/extra_script/modules.csv -# and let this template with only 'base' module. - - - -## Without postgres optimization -2022-07-13 19:42 -2022-07-13 21:20 - -Duration : 1:37 (107) - -## With postgres optimization - -2022-07-13 21:52 -2022-07-14 23:11 - -duration : 1:19 (79) - -16% diff --git a/README.bk b/README.bk new file mode 100644 index 0000000..53d0525 --- /dev/null +++ b/README.bk @@ -0,0 +1,449 @@ + + + +# Usage + +**Note:** + +the term ``odoo-openupgrade-wizard`` can be replaced by ``oow`` +in all the command lines below. + + + +## Command: ``init`` + +```shell +odoo-openupgrade-wizard init\ + --initial-version=10.0\ + --final-version=12.0\ + --project-name=my-customer-10-12\ + --extra-repository=OCA/web,OCA/server-tools +``` + +Initialize a folder to make a migration from a 10.0 and a 12.0 database. +This will generate the following structure : + +``` +filestore/ +log/ + 2022_03_25__23_12_41__init.log + ... +postgres_data/ +scripts/ + step_1__update__10.0/ + pre-migration.sql + post-migration.py + step_2__upgrade__11.0/ + ... + step_3__upgrade__12.0/ + ... + step_4__update__12.0/ + ... +src/ + env_10.0/ + extra_debian_requirements.txt + Dockerfile + odoo.conf + extra_python_requirements.txt + repos.yml + src/ + env_11.0/ + ... + env_12.0/ + ... +config.yml +modules.csv +``` + +- ``config.yml`` is the main configuration file of your project. + +- ``modules.csv`` file is an optional file. You can fill it with the list + of your modules installed on your production. The first column of this + file should contain the technical name of the module. + +- ``log`` folder will contains all the log of the ``odoo-openupgrade-wizard`` + and the logs of the odoo instance that will be executed. + +- ``filestore`` folder will contains the filestore of the odoo database(s) + +- ``postgres_data`` folder will be used by postgres docker image to store + database. + +- ``scripts`` folder contains a folder per migration step. In each step folder: + - ``pre-migration.sql`` can contains extra SQL queries you want to execute + before beginning the step. + - ``post-migration.py`` can contains extra python command to execute + after the execution of the step. + Script will be executed with ``odoo shell`` command. All the ORM is available + via the ``env`` variable. + +- ``src`` folder contains a folder per Odoo version. In each environment folder: + - ``repos.yml`` enumerates the list of the repositories to use to run the odoo instance. + The syntax should respect the ``gitaggregate`` command. + (See : https://pypi.org/project/git-aggregator/) + Repo files are pre-generated. You can update them with your custom settings. + (custom branches, extra PRs, git shallow options, etc...) + - ``extra_python_requirements.txt`` enumerates the list of extra python librairies + required to run the odoo instance. + The syntax should respect the ``pip install -r`` command. + (See : https://pip.pypa.io/en/stable/reference/requirements-file-format/) + - ``extra_debian_requirements.txt`` enumerates the list of extra system librairies + required to run the odoo instance. + - ``odoo.conf`` file. Add here extra configuration required for your custom modules. + the classical keys (``db_host``, ``db_port``, etc...) are automatically + autogenerated. + +At this step, you should change the autogenerated files. +You can use default files, if you have a very simple odoo instance without custom code, +extra repositories, or dependencies... + +**Note:** + +- In your repos.yml, preserve ``openupgrade`` and ``server-tools`` repositories + to have all the features of the librairies available. +- In your repos.yml file, the odoo project should be in ``./src/odoo`` + and the openupgrade project should be in ``./src/openupgrade/`` folder. + + + +## Command: ``pull-submodule`` + +**Prerequites:** init + being in a git repository. (if not, you can simply run ``git init``) + +if you already have a repos.yml file on github / gitlab, it can be convenient to +synchronize the repository, instead of copy past the ``repos.yml`` manually. + +In that case, you can add extra values, in the ``config.yml`` file in the section + +```yaml +odoo_version_settings: + 12.0: + repo_url: url_of_the_repo_that_contains_a_repos_yml_file + repo_branch: 12.0 + repo_file_path: repos.yml +``` + +then run following command : + +```shell +odoo-openupgrade-wizard pull-submodule +``` + + + +## Command: ``get-code`` + +**Prerequites:** init + +```shell +odoo-openupgrade-wizard get-code +``` + +This command will simply get all the Odoo code required to run all the steps +for your migration with the ``gitaggregate`` tools. + +The code is defined in the ``repos.yml`` of each environment folders. (or in the +directory ``repo_submodule`` if you use ``pull-submodule`` feature.) + +**Note** + +This step could take a big while ! + +**Optional arguments** + +if you want to update the code of some given versions, you can provide an extra parameter: + +```shell +odoo-openupgrade-wizard get-code --versions 10.0,11.0 +``` + + + +## Command: ``docker-build`` + +**Prerequites:** init + get-code + +This will build local docker images that will be used in the following steps. + +At this end of this step executing the following command should show a docker image per version. + + +```shell +docker images --filter "reference=odoo-openupgrade-wizard-*" +``` +``` +REPOSITORY TAG IMAGE ID CREATED SIZE +odoo-openupgrade-wizard-image---my-customer-10-12---12.0 latest ef664c366208 2 weeks ago 1.39GB +odoo-openupgrade-wizard-image---my-customer-10-12---11.0 latest 24e283fe4ae4 2 weeks ago 1.16GB +odoo-openupgrade-wizard-image---my-customer-10-12---10.0 latest 9d94dce2bd4e 2 weeks ago 924MB +``` + +**Optional arguments** + +- if you want to (re)build an image for some given versions, you can provide + an extra parameter: ``--versions 10.0,12.0`` + +**Note** + +- This step could take a big while also ! + + + + +## Command: ``run`` + +**Prerequites:** init + get-code + build + +```shell +odoo-openupgrade-wizard run\ + --step 1\ + --database DB_NAME +``` + +Run an Odoo instance with the environment defined by the step argument. + +The database will be created, if it doesn't exists. + +if ``stop-after-init`` is disabled, the odoo instance will be available +at your host, at the following url : http://localhost:9069 +(Port depends on your ``host_odoo_xmlrpc_port`` setting of your ``config.yml`` file) + +**Optional arguments** + +- You can add ``--init-modules=purchase,sale`` to install modules. + +- You can add ``stop-after-init`` flag to turn off the process at the end + of the installation. + + + + +## Command: ``install-from-csv`` + +**Prerequites:** init + get-code + build + +```shell +odoo-openupgrade-wizard install-from-csv\ + --database DB_NAME +``` + +Install the list of the modules defined in your ``modules.csv`` files on the +given database. + +The database will be created, if it doesn't exists. + +To get a correct ``modules.csv`` file, the following query can be used: +```shell +psql -c "copy (select name, shortdesc from ir_module_module where state = 'installed' order by 1) to stdout csv" coopiteasy +``` + + + + +## Command: ``upgrade`` + +**Prerequites:** init + get-code + build + +```shell +odoo-openupgrade-wizard upgrade\ + --database DB_NAME +``` + +Realize an upgrade of the database from the initial version to +the final version, following the different steps. + +For each step, it will : + +1. Execute the ``pre-migration.sql`` of the step. +2. Realize an "update all" (in an upgrade or update context) +3. Execute the scripts via XML-RPC (via ``odoorpc``) defined in + the ``post-migration.py`` file. + +**Optional arguments** + +- You can add ``--first-step=2`` to start at the second step. + +- You can add ``--last-step=3`` to end at the third step. + + + + +## Command: ``generate-module-analysis`` + +**Prerequites:** init + get-code + build + +```shell +odoo-openupgrade-wizard generate-module-analysis\ + --database DB_NAME + --step 2 + --modules MODULE_LIST +``` + +Realize an analyze between the target version (in parameter via the step argument) +and the previous version. It will generate analysis_file.txt files present +in OpenUpgrade project. +You can also use this fonction to analyze differences for custom / OCA modules +between several versions, in case of refactoring. + + + +## Command: ``estimate-workload`` + +**Prerequites:** init + get-code + +```shell +odoo-openupgrade-wizard estimate-workload +``` + +Generate an HTML file name ``analysis.html`` with all the information regarding +the work to do for the migration. +- checks that the modules are present in each version. (by managing the + renaming or merging of modules) +- check that the analysis and migration have been done for the official + modules present in odoo/odoo + + + +## Command: ``psql`` + +**Prerequites:** init + +```shell +odoo-openupgrade-wizard psql + --database DB_NAME + --command "SQL_REQUEST" +``` + +Execute an SQL Request on the target database. + +**Optional arguments** + +- If no ``database`` is provided, default ``postgres`` database will be used. exemple: + +```shell +odoo-openupgrade-wizard psql --command "\l"; +``` +Result: +``` + List of databases + Name | Owner | Encoding | Collate | Ctype | Access privileges +------------+-------+----------+------------+------------+------------------- + postgres | odoo | UTF8 | en_US.utf8 | en_US.utf8 | + template0 | odoo | UTF8 | en_US.utf8 | en_US.utf8 | =c/odoo + + | | | | | odoo=CTc/odoo + template1 | odoo | UTF8 | en_US.utf8 | en_US.utf8 | =c/odoo + + | | | | | odoo=CTc/odoo + test_psql | odoo | UTF8 | en_US.utf8 | en_US.utf8 | + +``` + +- if you execute request that return long result, you can choose to select ``pager`` or ``-no-pager`` + option to display the result via the click function ``echo_via_pager``. + (see : https://click.palletsprojects.com/en/8.1.x/utils/#pager-support) + +Note : Pager is enabled by default. + + +- you can pass extra psql arguments inline. + +```shell +odoo-openupgrade-wizard psql + --database=test_psql + --command "select id, name from res_partner where name ilike '%admin%';" + -H +``` +Result: +```html + + + + + + + + + +
idname
3Administrator
+

(1 row)
+

+ +``` + +See all the options here https://www.postgresql.org/docs/current/app-psql.html + + + +## Command: ``copydb`` + +**Prerequites:** init + +```shell +odoo-openupgrade-wizard copydb + --source DB_NAME + --dest NEW_DB_NAME +``` + +Create an Odoo database by copying an existing one. + +This script copies using postgres CREATEDB WITH TEMPLATE. It also copies +the filestore. + + + +## Command: ``dropdb`` + +**Prerequites:** init + +```shell +odoo-openupgrade-wizard dropdb + --database DB_NAME +``` + +Delete an Odoo database and its filestore. + +This command will always success even if DB_NAME does not exists. + + + +## Command: ``dumpdb`` + +**Prerequites:** init + +```shell +odoo-openupgrade-wizard dumpdb + --database DB_NAME + --database-path DATABASE_PATH + --filestore-path FILESTORE_PATH +``` + +Dump the database DB_NAME to DATABASE_PATH and export the filestore +related to DB_NAME into FILESTORE_PATH. To choose the format of the +backup files look at the `--database-format` and `--filestore-format`. + +*WARNING*: DATABASE_PATH should be a sub directory of the project path +in orter to have the postgresql container able to write the dump file. +For example, the project path is `/path/to/myproject` (where you run the +`init` command), then DATABASE_PATH can be any of the subdirectory of +`/path/to/myproject`. + +**Optional arguments** + +- To chose the database format use `--database-format`. Format can be + one of the following: + - `p` for plain sql text + - `c` for custom compressed backup of `pg_dump` + - `d` for directory structure + - `t` for a tar version of the directory structure + See also https://www.postgresql.org/docs/current/app-pgdump.html + The default database format is `c`. + +- To chose the filestore format use `--filestore-format`. Format can be + one of the following: + - `d` copy of the directory structure + - `t` tar version of the directory structure (not compressed) + - `tgz` tar version of the directory structure compressed with gzip. + The default filestore format is `tgz`. + +- By default, if database file or filestore file already exists, the + command will fail, preserving the existing dump. If you need to + overwrite the existing files, the `--force` option can be used. diff --git a/README.md b/README.md index e3cb191..9739a4d 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ + + [![Gitlab CI](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/badges/main/pipeline.svg)](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/-/pipelines) [![codecov](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/badges/main/coverage.svg)](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/) [![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0) @@ -11,9 +13,10 @@ Odoo Openupgrade Wizard is a tool that helps developpers to make major upgrade of Odoo Community Edition. (formely OpenERP). -It works with Openupgrade OCA tools. (https://github.com/oca/openupgrade) +It works with Openupgrade OCA tools. () this tool is useful for complex migrations: + - migrate several versions - take advantage of the migration to install / uninstall modules - execute sql requests or odoo shell scripts between each migration @@ -22,37 +25,45 @@ this tool is useful for complex migrations: It will create a migration environment (with all the code available) and provides helpers to run (and replay) migrations until it works. -* To develop and contribute to the library, refer to the ``DEVELOP.md`` file. -* Refer to the ``ROADMAP.md`` file to see the current limitation, bugs, and task to do. -* See authors in the ``CONTRIBUTORS.md`` file. +- To develop and contribute to the library, refer to the + [DEVELOP.md](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/-/blob/main/DEVELOP.md) + file. +- Refer to the + [Gitlab Issue Page](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/-/issues) + to see the current limitation, bugs, and task to do. +- See authors in the + [CONTRIBUTORS.md](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/-/blob/main/CONTRIBUTORS.md) + file. +- See history in the + [CHANGES.rst](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/-/blob/main/CHANGES.rst) + file. -# Table of Contents +## Table of Contents -* [Installation](#installation) -* [Usage](#usage) - * [Command ``init``](#command-init) - * [Command ``pull-submodule``](#command-pull-submodule) - * [Command ``get-code``](#command-get-code) - * [Command ``docker-build``](#command-docker-build) - * [Command ``run``](#command-run) - * [Command ``install-from-csv``](#command-install-from-csv) - * [Command ``upgrade``](#command-upgrade) - * [Command ``generate-module-analysis``](#command-generate-module-analysis) - * [Command ``estimate-workload``](#command-estimate-workload) - * [Command ``psql``](#command-psql) - * [Command ``copydb``](#command-copydb) - * [Command ``dropdb``](#command-dropdb) - * [Command ``dumpdb``](#command-dumpdb) +- [Installation](#installation) +- [Usage](#usage) + - [Command ``init``](#command-init) + - [Command ``init``](#command-init) + - [Command ``pull-submodule``](#command-pull-submodule) + - [Command ``get-code``](#command-get-code) + - [Command ``docker-build``](#command-docker-build) + - [Command ``run``](#command-run) + - [Command ``install-from-csv``](#command-install-from-csv) + - [Command ``upgrade``](#command-upgrade) + - [Command ``generate-module-analysis``](#command-generate-module-analysis) + - [Command ``estimate-workload``](#command-estimate-workload) + - [Command ``psql``](#command-psql) + - [Command ``copydb``](#command-copydb) + - [Command ``dropdb``](#command-dropdb) + - [Command ``dumpdb``](#command-dumpdb) - - -# Installation +## Installation **Prerequites:** -* The tools run on debian system -* You should have docker installed on your system -* Some features require extra packages. To have all the features available run: +- The tools run on debian system +- You should have docker installed on your system +- Some features require extra packages. To have all the features available run: **Installation:** @@ -60,459 +71,10 @@ The library is available on [PyPI](https://pypi.org/project/odoo-openupgrade-wiz To install it simply run : -``pipx install odoo-openupgrade-wizard`` - -(See alternative installation in ``DEVELOP.md`` file.) - - - -# Usage - -**Note:** - -the term ``odoo-openupgrade-wizard`` can be replaced by ``oow`` -in all the command lines below. - - - -## Command: ``init`` - ```shell -odoo-openupgrade-wizard init\ - --initial-version=10.0\ - --final-version=12.0\ - --project-name=my-customer-10-12\ - --extra-repository=OCA/web,OCA/server-tools +pipx install odoo-openupgrade-wizard ``` -Initialize a folder to make a migration from a 10.0 and a 12.0 database. -This will generate the following structure : - -``` -filestore/ -log/ - 2022_03_25__23_12_41__init.log - ... -postgres_data/ -scripts/ - step_1__update__10.0/ - pre-migration.sql - post-migration.py - step_2__upgrade__11.0/ - ... - step_3__upgrade__12.0/ - ... - step_4__update__12.0/ - ... -src/ - env_10.0/ - extra_debian_requirements.txt - Dockerfile - odoo.conf - extra_python_requirements.txt - repos.yml - src/ - env_11.0/ - ... - env_12.0/ - ... -config.yml -modules.csv -``` - -* ``config.yml`` is the main configuration file of your project. - -* ``modules.csv`` file is an optional file. You can fill it with the list - of your modules installed on your production. The first column of this - file should contain the technical name of the module. - -* ``log`` folder will contains all the log of the ``odoo-openupgrade-wizard`` - and the logs of the odoo instance that will be executed. - -* ``filestore`` folder will contains the filestore of the odoo database(s) - -* ``postgres_data`` folder will be used by postgres docker image to store - database. - -* ``scripts`` folder contains a folder per migration step. In each step folder: - - ``pre-migration.sql`` can contains extra SQL queries you want to execute - before beginning the step. - - ``post-migration.py`` can contains extra python command to execute - after the execution of the step. - Script will be executed with ``odoo shell`` command. All the ORM is available - via the ``env`` variable. - -* ``src`` folder contains a folder per Odoo version. In each environment folder: - - - ``repos.yml`` enumerates the list of the repositories to use to run the odoo instance. - The syntax should respect the ``gitaggregate`` command. - (See : https://pypi.org/project/git-aggregator/) - Repo files are pre-generated. You can update them with your custom settings. - (custom branches, extra PRs, git shallow options, etc...) - - - ``extra_python_requirements.txt`` enumerates the list of extra python librairies - required to run the odoo instance. - The syntax should respect the ``pip install -r`` command. - (See : https://pip.pypa.io/en/stable/reference/requirements-file-format/) - - - ``extra_debian_requirements.txt`` enumerates the list of extra system librairies - required to run the odoo instance. - - - ``odoo.conf`` file. Add here extra configuration required for your custom modules. - the classical keys (``db_host``, ``db_port``, etc...) are automatically - autogenerated. - -At this step, you should change the autogenerated files. -You can use default files, if you have a very simple odoo instance without custom code, -extra repositories, or dependencies... - -**Note:** - -- In your repos.yml, preserve ``openupgrade`` and ``server-tools`` repositories - to have all the features of the librairies available. -- In your repos.yml file, the odoo project should be in ``./src/odoo`` - and the openupgrade project should be in ``./src/openupgrade/`` folder. - - - -## Command: ``pull-submodule`` - -**Prerequites:** init + being in a git repository. (if not, you can simply run ``git init``) - -if you already have a repos.yml file on github / gitlab, it can be convenient to -synchronize the repository, instead of copy past the ``repos.yml`` manually. - -In that case, you can add extra values, in the ``config.yml`` file in the section - -```yaml -odoo_version_settings: - 12.0: - repo_url: url_of_the_repo_that_contains_a_repos_yml_file - repo_branch: 12.0 - repo_file_path: repos.yml -``` - -then run following command : - -```shell -odoo-openupgrade-wizard pull-submodule -``` - - - -## Command: ``get-code`` - -**Prerequites:** init - -```shell -odoo-openupgrade-wizard get-code -``` - -This command will simply get all the Odoo code required to run all the steps -for your migration with the ``gitaggregate`` tools. - -The code is defined in the ``repos.yml`` of each environment folders. (or in the -directory ``repo_submodule`` if you use ``pull-submodule`` feature.) - -**Note** - -* This step could take a big while ! - -**Optional arguments** - -if you want to update the code of some given versions, you can provide an extra parameter: - -```shell -odoo-openupgrade-wizard get-code --versions 10.0,11.0 -``` - - - -## Command: ``docker-build`` - -**Prerequites:** init + get-code - -This will build local docker images that will be used in the following steps. - -At this end of this step executing the following command should show a docker image per version. - - -```shell -docker images --filter "reference=odoo-openupgrade-wizard-*" -``` -``` -REPOSITORY TAG IMAGE ID CREATED SIZE -odoo-openupgrade-wizard-image---my-customer-10-12---12.0 latest ef664c366208 2 weeks ago 1.39GB -odoo-openupgrade-wizard-image---my-customer-10-12---11.0 latest 24e283fe4ae4 2 weeks ago 1.16GB -odoo-openupgrade-wizard-image---my-customer-10-12---10.0 latest 9d94dce2bd4e 2 weeks ago 924MB -``` - -**Optional arguments** - -* if you want to (re)build an image for some given versions, you can provide - an extra parameter: ``--versions 10.0,12.0`` - -**Note** - -* This step could take a big while also ! - - - - -## Command: ``run`` - -**Prerequites:** init + get-code + build - -```shell -odoo-openupgrade-wizard run\ - --step 1\ - --database DB_NAME -``` - -Run an Odoo instance with the environment defined by the step argument. - -The database will be created, if it doesn't exists. - -if ``stop-after-init`` is disabled, the odoo instance will be available -at your host, at the following url : http://localhost:9069 -(Port depends on your ``host_odoo_xmlrpc_port`` setting of your ``config.yml`` file) - -**Optional arguments** - -* You can add ``--init-modules=purchase,sale`` to install modules. - -* You can add ``stop-after-init`` flag to turn off the process at the end - of the installation. - - - - -## Command: ``install-from-csv`` - -**Prerequites:** init + get-code + build - -```shell -odoo-openupgrade-wizard install-from-csv\ - --database DB_NAME -``` - -Install the list of the modules defined in your ``modules.csv`` files on the -given database. - -The database will be created, if it doesn't exists. - -To get a correct ``modules.csv`` file, the following query can be used: -```shell -psql -c "copy (select name, shortdesc from ir_module_module where state = 'installed' order by 1) to stdout csv" coopiteasy -``` - - - - -## Command: ``upgrade`` - -**Prerequites:** init + get-code + build - -```shell -odoo-openupgrade-wizard upgrade\ - --database DB_NAME -``` - -Realize an upgrade of the database from the initial version to -the final version, following the different steps. - -For each step, it will : - -1. Execute the ``pre-migration.sql`` of the step. -2. Realize an "update all" (in an upgrade or update context) -3. Execute the scripts via XML-RPC (via ``odoorpc``) defined in - the ``post-migration.py`` file. - -**Optional arguments** - -* You can add ``--first-step=2`` to start at the second step. - -* You can add ``--last-step=3`` to end at the third step. - - - - -## Command: ``generate-module-analysis`` - -**Prerequites:** init + get-code + build - -```shell -odoo-openupgrade-wizard generate-module-analysis\ - --database DB_NAME - --step 2 - --modules MODULE_LIST -``` - -Realize an analyze between the target version (in parameter via the step argument) -and the previous version. It will generate analysis_file.txt files present -in OpenUpgrade project. -You can also use this fonction to analyze differences for custom / OCA modules -between several versions, in case of refactoring. - - - -## Command: ``estimate-workload`` - -**Prerequites:** init + get-code - -```shell -odoo-openupgrade-wizard estimate-workload -``` - -Generate an HTML file name ``analysis.html`` with all the information regarding -the work to do for the migration. -- checks that the modules are present in each version. (by managing the - renaming or merging of modules) -- check that the analysis and migration have been done for the official - modules present in odoo/odoo - - - -## Command: ``psql`` - -**Prerequites:** init - -```shell -odoo-openupgrade-wizard psql - --database DB_NAME - --command "SQL_REQUEST" -``` - -Execute an SQL Request on the target database. - -**Optional arguments** - -* If no ``database`` is provided, default ``postgres`` database will be used. exemple: - -```shell -odoo-openupgrade-wizard psql --command "\l"; -``` -Result: -``` - List of databases - Name | Owner | Encoding | Collate | Ctype | Access privileges -------------+-------+----------+------------+------------+------------------- - postgres | odoo | UTF8 | en_US.utf8 | en_US.utf8 | - template0 | odoo | UTF8 | en_US.utf8 | en_US.utf8 | =c/odoo + - | | | | | odoo=CTc/odoo - template1 | odoo | UTF8 | en_US.utf8 | en_US.utf8 | =c/odoo + - | | | | | odoo=CTc/odoo - test_psql | odoo | UTF8 | en_US.utf8 | en_US.utf8 | - -``` - -* if you execute request that return long result, you can choose to select ``pager`` or ``-no-pager`` - option to display the result via the click function ``echo_via_pager``. - (see : https://click.palletsprojects.com/en/8.1.x/utils/#pager-support) - -Note : Pager is enabled by default. - - -* you can pass extra psql arguments inline. - -```shell -odoo-openupgrade-wizard psql - --database=test_psql - --command "select id, name from res_partner where name ilike '%admin%';" - -H -``` -Result: -```html - - - - - - - - - -
idname
3Administrator
-

(1 row)
-

- -``` - -See all the options here https://www.postgresql.org/docs/current/app-psql.html - - - -## Command: ``copydb`` - -**Prerequites:** init - -```shell -odoo-openupgrade-wizard copydb - --source DB_NAME - --dest NEW_DB_NAME -``` - -Create an Odoo database by copying an existing one. - -This script copies using postgres CREATEDB WITH TEMPLATE. It also copies -the filestore. - - - -## Command: ``dropdb`` - -**Prerequites:** init - -```shell -odoo-openupgrade-wizard dropdb - --database DB_NAME -``` - -Delete an Odoo database and its filestore. - -This command will always success even if DB_NAME does not exists. - - - -## Command: ``dumpdb`` - -**Prerequites:** init - -```shell -odoo-openupgrade-wizard dumpdb - --database DB_NAME - --database-path DATABASE_PATH - --filestore-path FILESTORE_PATH -``` - -Dump the database DB_NAME to DATABASE_PATH and export the filestore -related to DB_NAME into FILESTORE_PATH. To choose the format of the -backup files look at the `--database-format` and `--filestore-format`. - -*WARNING*: DATABASE_PATH should be a sub directory of the project path -in orter to have the postgresql container able to write the dump file. -For example, the project path is `/path/to/myproject` (where you run the -`init` command), then DATABASE_PATH can be any of the subdirectory of -`/path/to/myproject`. - -**Optional arguments** - -* To chose the database format use `--database-format`. Format can be - one of the following: - - `p` for plain sql text - - `c` for custom compressed backup of `pg_dump` - - `d` for directory structure - - `t` for a tar version of the directory structure - See also https://www.postgresql.org/docs/current/app-pgdump.html - The default database format is `c`. - -* To chose the filestore format use `--filestore-format`. Format can be - one of the following: - - `d` copy of the directory structure - - `t` tar version of the directory structure (not compressed) - - `tgz` tar version of the directory structure compressed with gzip. - The default filestore format is `tgz`. - -* By default, if database file or filestore file already exists, the - command will fail, preserving the existing dump. If you need to - overwrite the existing files, the `--force` option can be used. +See alternative installation in the +[DEVELOP.md](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/-/blob/main/DEVELOP.md) +file. diff --git a/ROADMAP.md b/ROADMAP.md deleted file mode 100644 index 7e4bb00..0000000 --- a/ROADMAP.md +++ /dev/null @@ -1,45 +0,0 @@ -# Python Version - -* py310 is not available, due to dependencies to ``odoorpc`` that raise an error : - ``ERROR tests/cli_A_init_test.py - AttributeError: module 'collections' has no attribute 'MutableMapping'`` - Follow bug : https://stackoverflow.com/questions/69512672/getting-attributeerror-module-collections-has-no-attribute-mutablemapping-w - -# openUpgradelib Versions - -* ``openupgradelib`` requires a new feature psycopg2.sql since - (21 Aug 2019) - https://github.com/OCA/openupgradelib/commit/7408580e4469ba4b0cabb923da7facd71567a2fb - so we pin openupgradelib==2.0.0 (21 Jul 2018) - -The python version in the Odoo:12 docker image is : ``Python 3.5.3 (default, Apr 5 2021, 09:00:41)`` that is very old. - - -- https://github.com/OCA/openupgradelib/issues/248 -- https://github.com/OCA/openupgradelib/issues/288 -- https://github.com/OCA/openupgradelib.git@ed01555b8ae20f66b3af178c8ecaf6edd110ce75#egg=openupgradelib - -TODO : Fix via another way (other way than pining ``openuppgradelib`` version) the problem of old odoo versions. (it makes the upgrade failing for old revision (V8, etc...)) - -# Gitlab-CI - -* for the time being, Features requiring ``odoorpc`` are failing in gitlab-CI. - Tests are working locally but there is a network problem. For that reason, tests witch names - begins by ``cli_2`` like (``cli_20_install_from_csv_test.py``) are disabled in ``.gitlab-ci.yml``. - -TODO : work with Pierrick Brun, to run gitlab-runner on Akretion CI (without docker), to see if it is -fixing the problem. - -# Features Work In Progress - -* Add a tools to analyze workload. - -# Possible Improvments - -* select ``without-demo all`` depending on if the database - is created or not (and if current database contains demo data, checking if base.user_demo exists ?) - -# Other points not in the scope of GRAP work - -* Allow to use custom docker images. - -* Check if there are default values for containers, limiting ressources.