hobbyhimmel/odoo-openupgrade-wizard
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

wip

Browse Source
This commit is contained in:
Sylvain LE GAL 2024-11-05 17:25:10 +01:00
parent 8791a883f9
commit 9b2cd724b5
8 changed files with 572 additions and 647 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.markdownlint.rb Normal file
View File

@ -0,0 +1,4 @@
all
# MD033 Inline HTML
# TODO: Tweak elements when https://github.com/markdownlint/markdownlint/issues/118 will be done?
exclude_rule 'MD033'

8
.pre-commit-config.yaml
View File

@ -30,6 +30,14 @@ repos:
rev: "6.1.0" rev: "6.1.0"
hooks: hooks:
- id: flake8 - id: flake8
- repo: https://github.com/igorshubovych/markdownlint-cli
rev: "v0.41.0"
hooks:
- id: markdownlint
args: [ --disable=MD033 ]
# - repo: https://gitlab.com/smop/pre-commit-hooks # - repo: https://gitlab.com/smop/pre-commit-hooks
# rev: v1.0.0 # rev: v1.0.0
# hooks: # hooks:

14
CONTRIBUTORS.md
View File

@ -1,4 +1,6 @@
# Developers # Contributors
## Developers
* Sylvain LE GAL, from [GRAP](http://www.grap.coop), since March 2022 * Sylvain LE GAL, from [GRAP](http://www.grap.coop), since March 2022
* Rémy TAYMANS, from [Coop It Easy](https://coopiteasy.be/), since June 2022 * Rémy TAYMANS, from [Coop It Easy](https://coopiteasy.be/), since June 2022
@ -9,9 +11,11 @@
* Hugues DE KEYSER, from [Coop It Easy](https://coopiteasy.be/), since April 2024 * Hugues DE KEYSER, from [Coop It Easy](https://coopiteasy.be/), since April 2024
* Gabriel PICKENHAYN, since May 2024 * Gabriel PICKENHAYN, since May 2024
* Boris GALLET, since September 2024 * Boris GALLET, since September 2024
* Ahmet YIĞIT BUDAK, from [Altinkaya](https://www.altinkaya.com/fr), since October 2024 * Ahmet YIĞIT BUDAK, from [Altinkaya](https://www.altinkaya.com/fr),
* Alexandre AUBIN, from [Coopaname](https://www.coopaname.coop/), since October 2024 since October 2024
* Alexandre AUBIN, from [Coopaname](https://www.coopaname.coop/),
since October 2024
# Reviewers ## Reviewers
* Sébastien BEAU, from Akretion (https://akretion.com) * Sébastien BEAU, from [Akretion](https://akretion.com)

110
DEVELOP.md
View File

@ -1,8 +1,10 @@
# Installation to develop # Develop
## Basic installation ## Installation to develop
``` ### Basic installation
```shell
git clone https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/ git clone https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/
cd odoo-openupgrade-wizard cd odoo-openupgrade-wizard
virtualenv env --python=python3 virtualenv env --python=python3
@ -12,19 +14,20 @@ poetry install
``odoo-openupgrade-wizard`` commands are now available in your virutalenv. ``odoo-openupgrade-wizard`` commands are now available in your virutalenv.
## Advanced installation ### Advanced installation
If you want to use this library without installing anything in your If you want to use this library without installing anything in your
system, execute the following steps, otherwise, go to 'Installation' part. system, execute the following steps, otherwise, go to 'Installation' part.
1. Run a docker container: - Run a docker container:
``docker run -it ubuntu:focal``
2. Execute the following commnands
```shell
docker run -it ubuntu:focal
``` ```
- Execute the following commnands
```shell
apt-get update apt-get update
apt-get install git python3 python3-pip python3-venv apt-get install git python3 python3-pip python3-venv
@ -37,40 +40,40 @@ pipx install virtualenv
pipx install poetry pipx install poetry
``` ```
# Run tests ## Run tests
## Via pytest (simple) ### Via pytest (simple)
This will run tests only for the current ``python3.X`` version. This will run tests only for the current ``python3.X`` version.
(in your virtualenv) (in your virtualenv)
```
```shell
poetry run pytest --cov odoo_openupgrade_wizard --verbosity=2 --exitfirst poetry run pytest --cov odoo_openupgrade_wizard --verbosity=2 --exitfirst
``` ```
## Via Tox (advanced) ### Via Tox (advanced)
This will run tests for all the python versions put in the ``tox.ini`` folder. This will run tests for all the python versions put in the ``tox.ini`` folder.
(in your virtualenv) (in your virtualenv)
```
```shell
tox tox
``` ```
Note : you should have all the python versions available in your local system. Note : you should have all the python versions available in your local system.
```shell
```
sudo apt-get install python3.6 python3.6-distutils sudo apt-get install python3.6 python3.6-distutils
sudo apt-get install python3.7 python3.7-distutils sudo apt-get install python3.7 python3.7-distutils
sudo apt-get install python3.8 python3.8-distutils sudo apt-get install python3.8 python3.8-distutils
sudo apt-get install python3.9 python3.9-distutils sudo apt-get install python3.9 python3.9-distutils
``` ```
## Via Gitlab Runner locally ### Via Gitlab Runner locally
```shell
```
# Install tools # Install tools
pipx install gitlabci-local pipx install gitlabci-local
@ -78,17 +81,19 @@ pipx install gitlabci-local
gitlabci-local gitlabci-local
``` ```
# Debugging ## Debugging
Some docker command could help you when using / developping this tools. Some docker command could help you when using / developping this tools.
**Enter the postgres container** To Enter the postgres container, run:
```shell
docker exec -it POSTGRES_CONTAINER_NAME /bin/bash docker exec -it POSTGRES_CONTAINER_NAME /bin/bash
```
# Contribute ## Contribute
## Provide newsfragments in your merge requests ### Provide newsfragments in your merge requests
If you propose a merge request, please add a newsfragments with it. If you propose a merge request, please add a newsfragments with it.
@ -115,11 +120,12 @@ a `+` symbol.
This is a example of newsfragments. This is a example of newsfragments.
`newsfragments/+sub-command-cowsay.feature`: `newsfragments/+sub-command-cowsay.feature`:
``` ```
Adds a new subcommand `cowsay` to allow poeple to speak like a cow. Adds a new subcommand `cowsay` to allow poeple to speak like a cow.
``` ```
## Add python dependencies ### Add python dependencies
If you add new dependencies, you have to: If you add new dependencies, you have to:
@ -127,7 +133,7 @@ If you add new dependencies, you have to:
- run the following command in your virtualenv : ``poetry update`` - run the following command in your virtualenv : ``poetry update``
## Release on Gitlab and publish on PyPI ### Release on Gitlab and publish on PyPI
Building, releasing and publishing a new version works with tags. Building, releasing and publishing a new version works with tags.
@ -143,7 +149,7 @@ on gitlab, but they trigger a publication on PyPI.
Before creating a tag, ensure that the version of the program is Before creating a tag, ensure that the version of the program is
updated. To update the program version you can use the command: updated. To update the program version you can use the command:
``` ```shell
poetry version {major,minor,patch} poetry version {major,minor,patch}
``` ```
@ -163,27 +169,26 @@ To see if the publication on PyPI and the release on gitlab were done
correctly, go in Build > Pipelines. You will find a pipeline for the tag correctly, go in Build > Pipelines. You will find a pipeline for the tag
you just created. you just created.
## Understanding the library
# Understanding the library ### Tools to understand
## Tools to understand
The library is using many tools. It is recommanded to understand that tools The library is using many tools. It is recommanded to understand that tools
to contribute to that project: to contribute to that project:
* Docker (https://www.docker.com/) - Docker (<https://www.docker.com/>)
* Gitlab CI (https://docs.gitlab.com/ee/ci/quick_start/index.html) - Gitlab CI (<https://docs.gitlab.com/ee/ci/quick_start/index.html>)
* openupgrade project (https://github.com/oca/openupgrade) and related openupgradelib (https://github.com/oca/openupgradelib) - openupgrade project (<https://github.com/oca/openupgrade>)
* poetry (https://python-poetry.org/) and related openupgradelib (<https://github.com/oca/openupgradelib>)
* odoorpc (https://github.com/OCA/odoorpc) - poetry (<https://python-poetry.org/>)
* git-aggregator (https://github.com/acsone/git-aggregator) - odoorpc (<https://github.com/OCA/odoorpc>)
- git-aggregator (<https://github.com/acsone/git-aggregator>)
Also this project is inspired by the following tools: Also this project is inspired by the following tools:
* click-odoo-contrib (https://github.com/acsone/click-odoo-contrib) - click-odoo-contrib (<https://github.com/acsone/click-odoo-contrib>)
## Dockerfile information
# Dockerfile information
### From version 5 to 7 ### From version 5 to 7
@ -195,61 +200,68 @@ Try to create dockerfile, based on the odoo official ones fails. Any help welcom
### From version 11.0 to latest version. (Python3) ### From version 11.0 to latest version. (Python3)
The Dockerfile of the version 11 to the lastest version of Odoo are written this way : The Dockerfile of the version 11 to the lastest version of Odoo are written
this way:
- Copy the content of https://github.com/odoo/odoo/blob/ODOO_VERSION/setup/package.dfsrc - Copy the content of <https://github.com/odoo/odoo/blob/ODOO_VERSION/setup/package.dfsrc>
- remove all the part after the big ``apt-get install`` - remove all the part after the big ``apt-get install``
- install debian package ``git`` to have the possibility to pip install from git url. - install debian package ``git`` to have the possibility
to pip install from git url.
- install custom debian packages - install custom debian packages
- install python odoo requirements - install python odoo requirements
- install python ``setuptools-scm`` lib to have the possibility to pip install ``openupgradelib`` from git url. - install python ``setuptools-scm`` lib to have the possibility
to pip install ``openupgradelib`` from git url.
- install python custom requirements - install python custom requirements
- makes link between external user and docker odoo user - makes link between external user and docker odoo user
## Réferences ## References
- how to install gitlab runner locally: - how to install gitlab runner locally:
https://docs.gitlab.com/runner/install/linux-manually.html <https://docs.gitlab.com/runner/install/linux-manually.html>
- Check your CI locally. (French) - Check your CI locally. (French)
https://blog.stephane-robert.info/post/gitlab-valider-ci-yml/ <https://blog.stephane-robert.info/post/gitlab-valider-ci-yml/>
https://blog.callr.tech/building-docker-images-with-gitlab-ci-best-practices/
<https://blog.callr.tech/building-docker-images-with-gitlab-ci-best-practices/>
## Python version settings depending on the debian version ## Python version settings depending on the debian version
This part can help you if you want to change your autogenerated Dockerfiles. This part can help you if you want to change your autogenerated Dockerfiles.
See (https://github.com/odoo/odoo/blob/ODOO_VERSION/setup/package.dfdebian) See (<https://github.com/odoo/odoo/blob/ODOO_VERSION/setup/package.dfdebian>).
### debian:wheezy (V7) (for Odoo: 8.0) ### debian:wheezy (V7) (for Odoo: 8.0)
- Ubuntu release : 12.04 xxx, 12.10 xxx, 13.04 xxx, 14.10 xxx - Ubuntu release : 12.04 xxx, 12.10 xxx, 13.04 xxx, 14.10 xxx
- python2.7 - python2.7
- First release : 04/05/2013 - First release : 04/05/2013
- End LTS : May 2018 - End LTS : May 2018
### debian:jessie (V8) (for Odoo: 9.0, 10.0) ### debian:jessie (V8) (for Odoo: 9.0, 10.0)
- Ubuntu release : 14.04 trusty, 14.10 utopic, 15.04 vivid, 15.10 wily - Ubuntu release : 14.04 trusty, 14.10 utopic, 15.04 vivid, 15.10 wily
- python2.7 - python2.7
- First release : 26/04/2015 - First release : 26/04/2015
- End LTS : June 2020 - End LTS : June 2020
### debian:stretch (V9) (for Odoo: 11.0, 12.0) ### debian:stretch (V9) (for Odoo: 11.0, 12.0)
- Ubuntu releases : 16.04 xenial, 16.10 yakkety, 17.04 zesty, 17.10 artful - Ubuntu releases : 16.04 xenial, 16.10 yakkety, 17.04 zesty, 17.10 artful
- python2.7 and python3.5 - python2.7 and python3.5
- First release : 17/06/2017 - First release : 17/06/2017
- End LTS : June 2022 - End LTS : June 2022
### debian:buster (13.0, 14.0) ### debian:buster (13.0, 14.0)
- Ubuntu release : 18.04 bionic, 18.10 cosmic, 19.04 disco, 19.10 eoan - Ubuntu release : 18.04 bionic, 18.10 cosmic, 19.04 disco, 19.10 eoan
- python2.7 and python3.7 - python2.7 and python3.7
- First release : 06/07/2019 - First release : 06/07/2019
- End LTS : Undefined. - End LTS : Undefined.
## debian:bullseye (15.0, 16.0) ### debian:bullseye (15.0, 16.0)
- Ubuntu release : 20.04 focal, 20.10 groovy, 21.04 hirsute, 21.10 impish - Ubuntu release : 20.04 focal, 20.10 groovy, 21.04 hirsute, 21.10 impish
- python3.9 - python3.9
- First release : 14/07/2021 - First release : 14/07/2021

69
INTERNAL_NOTES.md
View File

@ -1,69 +0,0 @@
# TODO
* with coop it easy :
- short_help of group decorator ? seems useless...
* add constrains on ``--step`` option.
* revert : set 777 to log and filestore to be able to write on this folder
inside the containers. TODO, ask to coop it easy or commown for better alternative.
* allow to call odoo-bin shell, via : https://github.com/d11wtq/dockerpty
(see https://github.com/docker/docker-py/issues/247)
# List of the series of odoo
# python version is defined, based on the OCA CI.
# https://github.com/OCA/oca-addons-repo-template/blob/master/src/.github/workflows/%7B%25%20if%20ci%20%3D%3D%20'GitHub'%20%25%7Dtest.yml%7B%25%20endif%20%25%7D.jinja
# tips
```
# execute sql request in postgres docker
docker exec db psql --username=odoo --dbname=test_v12 -c "update res_partner set ""email"" = 'bib@bqsdfqsdf.txt';"
```
# TODO Nice To have
- Fix gitlabci-local. For the time being, it is not possible to debug
locally. (there are extra bugs locally that doesn't occures on gitlab,
in ``cli_B_03_run_test.py``...
- add
# Try gitlab runner
curl -LJO "https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_amd64.deb"
sudo dpkg -i gitlab-runner_amd64.deb
(https://docs.gitlab.com/runner/install/linux-manually.html)
# TODO:
- check dynamic user id with
https://github.com/camptocamp/docker-odoo-project/blob/master/bin/docker-entrypoint.sh
in modules.csv.j2 :
# TODO, this value are usefull for test for analyse between 13 and 14.
# move that values in data/extra_script/modules.csv
# and let this template with only 'base' module.
## Without postgres optimization
2022-07-13 19:42
2022-07-13 21:20
Duration : 1:37 (107)
## With postgres optimization
2022-07-13 21:52
2022-07-14 23:11
duration : 1:19 (79)
16%

449
README.bk Normal file
View File

@ -0,0 +1,449 @@
<a name="usage"/>
# Usage
**Note:**
the term ``odoo-openupgrade-wizard`` can be replaced by ``oow``
in all the command lines below.
<a name="command-init"/>
## Command: ``init``
```shell
odoo-openupgrade-wizard init\
--initial-version=10.0\
--final-version=12.0\
--project-name=my-customer-10-12\
--extra-repository=OCA/web,OCA/server-tools
```
Initialize a folder to make a migration from a 10.0 and a 12.0 database.
This will generate the following structure :
```
filestore/
log/
2022_03_25__23_12_41__init.log
...
postgres_data/
scripts/
step_1__update__10.0/
pre-migration.sql
post-migration.py
step_2__upgrade__11.0/
...
step_3__upgrade__12.0/
...
step_4__update__12.0/
...
src/
env_10.0/
extra_debian_requirements.txt
Dockerfile
odoo.conf
extra_python_requirements.txt
repos.yml
src/
env_11.0/
...
env_12.0/
...
config.yml
modules.csv
```
- ``config.yml`` is the main configuration file of your project.
- ``modules.csv`` file is an optional file. You can fill it with the list
of your modules installed on your production. The first column of this
file should contain the technical name of the module.
- ``log`` folder will contains all the log of the ``odoo-openupgrade-wizard``
and the logs of the odoo instance that will be executed.
- ``filestore`` folder will contains the filestore of the odoo database(s)
- ``postgres_data`` folder will be used by postgres docker image to store
database.
- ``scripts`` folder contains a folder per migration step. In each step folder:
- ``pre-migration.sql`` can contains extra SQL queries you want to execute
before beginning the step.
- ``post-migration.py`` can contains extra python command to execute
after the execution of the step.
Script will be executed with ``odoo shell`` command. All the ORM is available
via the ``env`` variable.
- ``src`` folder contains a folder per Odoo version. In each environment folder:
- ``repos.yml`` enumerates the list of the repositories to use to run the odoo instance.
The syntax should respect the ``gitaggregate`` command.
(See : https://pypi.org/project/git-aggregator/)
Repo files are pre-generated. You can update them with your custom settings.
(custom branches, extra PRs, git shallow options, etc...)
- ``extra_python_requirements.txt`` enumerates the list of extra python librairies
required to run the odoo instance.
The syntax should respect the ``pip install -r`` command.
(See : https://pip.pypa.io/en/stable/reference/requirements-file-format/)
- ``extra_debian_requirements.txt`` enumerates the list of extra system librairies
required to run the odoo instance.
- ``odoo.conf`` file. Add here extra configuration required for your custom modules.
the classical keys (``db_host``, ``db_port``, etc...) are automatically
autogenerated.
At this step, you should change the autogenerated files.
You can use default files, if you have a very simple odoo instance without custom code,
extra repositories, or dependencies...
**Note:**
- In your repos.yml, preserve ``openupgrade`` and ``server-tools`` repositories
to have all the features of the librairies available.
- In your repos.yml file, the odoo project should be in ``./src/odoo``
and the openupgrade project should be in ``./src/openupgrade/`` folder.
<a name="command-pull-submodule"/>
## Command: ``pull-submodule``
**Prerequites:** init + being in a git repository. (if not, you can simply run ``git init``)
if you already have a repos.yml file on github / gitlab, it can be convenient to
synchronize the repository, instead of copy past the ``repos.yml`` manually.
In that case, you can add extra values, in the ``config.yml`` file in the section
```yaml
odoo_version_settings:
12.0:
repo_url: url_of_the_repo_that_contains_a_repos_yml_file
repo_branch: 12.0
repo_file_path: repos.yml
```
then run following command :
```shell
odoo-openupgrade-wizard pull-submodule
```
<a name="command-get-code"/>
## Command: ``get-code``
**Prerequites:** init
```shell
odoo-openupgrade-wizard get-code
```
This command will simply get all the Odoo code required to run all the steps
for your migration with the ``gitaggregate`` tools.
The code is defined in the ``repos.yml`` of each environment folders. (or in the
directory ``repo_submodule`` if you use ``pull-submodule`` feature.)
**Note**
This step could take a big while !
**Optional arguments**
if you want to update the code of some given versions, you can provide an extra parameter:
```shell
odoo-openupgrade-wizard get-code --versions 10.0,11.0
```
<a name="command-docker-build"/>
## Command: ``docker-build``
**Prerequites:** init + get-code
This will build local docker images that will be used in the following steps.
At this end of this step executing the following command should show a docker image per version.
```shell
docker images --filter "reference=odoo-openupgrade-wizard-*"
```
```
REPOSITORY TAG IMAGE ID CREATED SIZE
odoo-openupgrade-wizard-image---my-customer-10-12---12.0 latest ef664c366208 2 weeks ago 1.39GB
odoo-openupgrade-wizard-image---my-customer-10-12---11.0 latest 24e283fe4ae4 2 weeks ago 1.16GB
odoo-openupgrade-wizard-image---my-customer-10-12---10.0 latest 9d94dce2bd4e 2 weeks ago 924MB
```
**Optional arguments**
- if you want to (re)build an image for some given versions, you can provide
an extra parameter: ``--versions 10.0,12.0``
**Note**
- This step could take a big while also !
<a name="command-run"/>
## Command: ``run``
**Prerequites:** init + get-code + build
```shell
odoo-openupgrade-wizard run\
--step 1\
--database DB_NAME
```
Run an Odoo instance with the environment defined by the step argument.
The database will be created, if it doesn't exists.
if ``stop-after-init`` is disabled, the odoo instance will be available
at your host, at the following url : http://localhost:9069
(Port depends on your ``host_odoo_xmlrpc_port`` setting of your ``config.yml`` file)
**Optional arguments**
- You can add ``--init-modules=purchase,sale`` to install modules.
- You can add ``stop-after-init`` flag to turn off the process at the end
of the installation.
<a name="command-install-from-csv"/>
## Command: ``install-from-csv``
**Prerequites:** init + get-code + build
```shell
odoo-openupgrade-wizard install-from-csv\
--database DB_NAME
```
Install the list of the modules defined in your ``modules.csv`` files on the
given database.
The database will be created, if it doesn't exists.
To get a correct ``modules.csv`` file, the following query can be used:
```shell
psql -c "copy (select name, shortdesc from ir_module_module where state = 'installed' order by 1) to stdout csv" coopiteasy
```
<a name="command-upgrade"/>
## Command: ``upgrade``
**Prerequites:** init + get-code + build
```shell
odoo-openupgrade-wizard upgrade\
--database DB_NAME
```
Realize an upgrade of the database from the initial version to
the final version, following the different steps.
For each step, it will :
1. Execute the ``pre-migration.sql`` of the step.
2. Realize an "update all" (in an upgrade or update context)
3. Execute the scripts via XML-RPC (via ``odoorpc``) defined in
the ``post-migration.py`` file.
**Optional arguments**
- You can add ``--first-step=2`` to start at the second step.
- You can add ``--last-step=3`` to end at the third step.
<a name="command-generate-module-analysis"/>
## Command: ``generate-module-analysis``
**Prerequites:** init + get-code + build
```shell
odoo-openupgrade-wizard generate-module-analysis\
--database DB_NAME
--step 2
--modules MODULE_LIST
```
Realize an analyze between the target version (in parameter via the step argument)
and the previous version. It will generate analysis_file.txt files present
in OpenUpgrade project.
You can also use this fonction to analyze differences for custom / OCA modules
between several versions, in case of refactoring.
<a name="command-estimate-workload"/>
## Command: ``estimate-workload``
**Prerequites:** init + get-code
```shell
odoo-openupgrade-wizard estimate-workload
```
Generate an HTML file name ``analysis.html`` with all the information regarding
the work to do for the migration.
- checks that the modules are present in each version. (by managing the
renaming or merging of modules)
- check that the analysis and migration have been done for the official
modules present in odoo/odoo
<a name="command-psql"/>
## Command: ``psql``
**Prerequites:** init
```shell
odoo-openupgrade-wizard psql
--database DB_NAME
--command "SQL_REQUEST"
```
Execute an SQL Request on the target database.
**Optional arguments**
- If no ``database`` is provided, default ``postgres`` database will be used. exemple:
```shell
odoo-openupgrade-wizard psql --command "\l";
```
Result:
```
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
------------+-------+----------+------------+------------+-------------------
postgres | odoo | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | odoo | UTF8 | en_US.utf8 | en_US.utf8 | =c/odoo +
| | | | | odoo=CTc/odoo
template1 | odoo | UTF8 | en_US.utf8 | en_US.utf8 | =c/odoo +
| | | | | odoo=CTc/odoo
test_psql | odoo | UTF8 | en_US.utf8 | en_US.utf8 |
```
- if you execute request that return long result, you can choose to select ``pager`` or ``-no-pager``
option to display the result via the click function ``echo_via_pager``.
(see : https://click.palletsprojects.com/en/8.1.x/utils/#pager-support)
Note : Pager is enabled by default.
- you can pass extra psql arguments inline.
```shell
odoo-openupgrade-wizard psql
--database=test_psql
--command "select id, name from res_partner where name ilike '%admin%';"
-H
```
Result:
```html
<table border="1">
<tr>
<th align="center">id</th>
<th align="center">name</th>
</tr>
<tr valign="top">
<td align="right">3</td>
<td align="left">Administrator</td>
</tr>
</table>
<p>(1 row)<br />
</p>
```
See all the options here https://www.postgresql.org/docs/current/app-psql.html
<a name="command-copydb"/>
## Command: ``copydb``
**Prerequites:** init
```shell
odoo-openupgrade-wizard copydb
--source DB_NAME
--dest NEW_DB_NAME
```
Create an Odoo database by copying an existing one.
This script copies using postgres CREATEDB WITH TEMPLATE. It also copies
the filestore.
<a name="command-dropdb"/>
## Command: ``dropdb``
**Prerequites:** init
```shell
odoo-openupgrade-wizard dropdb
--database DB_NAME
```
Delete an Odoo database and its filestore.
This command will always success even if DB_NAME does not exists.
<a name="command-dumpdb"/>
## Command: ``dumpdb``
**Prerequites:** init
```shell
odoo-openupgrade-wizard dumpdb
--database DB_NAME
--database-path DATABASE_PATH
--filestore-path FILESTORE_PATH
```
Dump the database DB_NAME to DATABASE_PATH and export the filestore
related to DB_NAME into FILESTORE_PATH. To choose the format of the
backup files look at the `--database-format` and `--filestore-format`.
*WARNING*: DATABASE_PATH should be a sub directory of the project path
in orter to have the postgresql container able to write the dump file.
For example, the project path is `/path/to/myproject` (where you run the
`init` command), then DATABASE_PATH can be any of the subdirectory of
`/path/to/myproject`.
**Optional arguments**
- To chose the database format use `--database-format`. Format can be
one of the following:
- `p` for plain sql text
- `c` for custom compressed backup of `pg_dump`
- `d` for directory structure
- `t` for a tar version of the directory structure
See also https://www.postgresql.org/docs/current/app-pgdump.html
The default database format is `c`.
- To chose the filestore format use `--filestore-format`. Format can be
one of the following:
- `d` copy of the directory structure
- `t` tar version of the directory structure (not compressed)
- `tgz` tar version of the directory structure compressed with gzip.
The default filestore format is `tgz`.
- By default, if database file or filestore file already exists, the
command will fail, preserving the existing dump. If you need to
overwrite the existing files, the `--force` option can be used.

520
README.md
View File

@ -1,3 +1,5 @@
<!-- markdownlint-disable MD033 -->
[![Gitlab CI](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/badges/main/pipeline.svg)](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/-/pipelines) [![Gitlab CI](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/badges/main/pipeline.svg)](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/-/pipelines)
[![codecov](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/badges/main/coverage.svg)](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/) [![codecov](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/badges/main/coverage.svg)](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/)
[![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0) [![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0)
@ -11,9 +13,10 @@
Odoo Openupgrade Wizard is a tool that helps developpers to make major Odoo Openupgrade Wizard is a tool that helps developpers to make major
upgrade of Odoo Community Edition. (formely OpenERP). upgrade of Odoo Community Edition. (formely OpenERP).
It works with Openupgrade OCA tools. (https://github.com/oca/openupgrade) It works with Openupgrade OCA tools. (<https://github.com/oca/openupgrade>)
this tool is useful for complex migrations: this tool is useful for complex migrations:
- migrate several versions - migrate several versions
- take advantage of the migration to install / uninstall modules - take advantage of the migration to install / uninstall modules
- execute sql requests or odoo shell scripts between each migration - execute sql requests or odoo shell scripts between each migration
@ -22,37 +25,45 @@ this tool is useful for complex migrations:
It will create a migration environment (with all the code available) It will create a migration environment (with all the code available)
and provides helpers to run (and replay) migrations until it works. and provides helpers to run (and replay) migrations until it works.
* To develop and contribute to the library, refer to the ``DEVELOP.md`` file. - To develop and contribute to the library, refer to the
* Refer to the ``ROADMAP.md`` file to see the current limitation, bugs, and task to do. [DEVELOP.md](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/-/blob/main/DEVELOP.md)
* See authors in the ``CONTRIBUTORS.md`` file. file.
- Refer to the
[Gitlab Issue Page](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/-/issues)
to see the current limitation, bugs, and task to do.
- See authors in the
[CONTRIBUTORS.md](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/-/blob/main/CONTRIBUTORS.md)
file.
- See history in the
[CHANGES.rst](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/-/blob/main/CHANGES.rst)
file.
# Table of Contents ## Table of Contents
* [Installation](#installation) - [Installation](#installation)
* [Usage](#usage) - [Usage](#usage)
* [Command ``init``](#command-init) - [Command ``init``](#command-init)
* [Command ``pull-submodule``](#command-pull-submodule) - [Command ``init``](#command-init)
* [Command ``get-code``](#command-get-code) - [Command ``pull-submodule``](#command-pull-submodule)
* [Command ``docker-build``](#command-docker-build) - [Command ``get-code``](#command-get-code)
* [Command ``run``](#command-run) - [Command ``docker-build``](#command-docker-build)
* [Command ``install-from-csv``](#command-install-from-csv) - [Command ``run``](#command-run)
* [Command ``upgrade``](#command-upgrade) - [Command ``install-from-csv``](#command-install-from-csv)
* [Command ``generate-module-analysis``](#command-generate-module-analysis) - [Command ``upgrade``](#command-upgrade)
* [Command ``estimate-workload``](#command-estimate-workload) - [Command ``generate-module-analysis``](#command-generate-module-analysis)
* [Command ``psql``](#command-psql) - [Command ``estimate-workload``](#command-estimate-workload)
* [Command ``copydb``](#command-copydb) - [Command ``psql``](#command-psql)
* [Command ``dropdb``](#command-dropdb) - [Command ``copydb``](#command-copydb)
* [Command ``dumpdb``](#command-dumpdb) - [Command ``dropdb``](#command-dropdb)
- [Command ``dumpdb``](#command-dumpdb)
<a name="installation"/> ## Installation<a name="installation"></a>
# Installation
**Prerequites:** **Prerequites:**
* The tools run on debian system - The tools run on debian system
* You should have docker installed on your system - You should have docker installed on your system
* Some features require extra packages. To have all the features available run: - Some features require extra packages. To have all the features available run:
**Installation:** **Installation:**
@ -60,459 +71,10 @@ The library is available on [PyPI](https://pypi.org/project/odoo-openupgrade-wiz
To install it simply run : To install it simply run :
``pipx install odoo-openupgrade-wizard``
(See alternative installation in ``DEVELOP.md`` file.)
<a name="usage"/>
# Usage
**Note:**
the term ``odoo-openupgrade-wizard`` can be replaced by ``oow``
in all the command lines below.
<a name="command-init"/>
## Command: ``init``
```shell ```shell
odoo-openupgrade-wizard init\ pipx install odoo-openupgrade-wizard
--initial-version=10.0\
--final-version=12.0\
--project-name=my-customer-10-12\
--extra-repository=OCA/web,OCA/server-tools
``` ```
Initialize a folder to make a migration from a 10.0 and a 12.0 database. See alternative installation in the
This will generate the following structure : [DEVELOP.md](https://gitlab.com/odoo-openupgrade-wizard/odoo-openupgrade-wizard/-/blob/main/DEVELOP.md)
file.
```
filestore/
log/
2022_03_25__23_12_41__init.log
...
postgres_data/
scripts/
step_1__update__10.0/
pre-migration.sql
post-migration.py
step_2__upgrade__11.0/
...
step_3__upgrade__12.0/
...
step_4__update__12.0/
...
src/
env_10.0/
extra_debian_requirements.txt
Dockerfile
odoo.conf
extra_python_requirements.txt
repos.yml
src/
env_11.0/
...
env_12.0/
...
config.yml
modules.csv
```
* ``config.yml`` is the main configuration file of your project.
* ``modules.csv`` file is an optional file. You can fill it with the list
of your modules installed on your production. The first column of this
file should contain the technical name of the module.
* ``log`` folder will contains all the log of the ``odoo-openupgrade-wizard``
and the logs of the odoo instance that will be executed.
* ``filestore`` folder will contains the filestore of the odoo database(s)
* ``postgres_data`` folder will be used by postgres docker image to store
database.
* ``scripts`` folder contains a folder per migration step. In each step folder:
- ``pre-migration.sql`` can contains extra SQL queries you want to execute
before beginning the step.
- ``post-migration.py`` can contains extra python command to execute
after the execution of the step.
Script will be executed with ``odoo shell`` command. All the ORM is available
via the ``env`` variable.
* ``src`` folder contains a folder per Odoo version. In each environment folder:
- ``repos.yml`` enumerates the list of the repositories to use to run the odoo instance.
The syntax should respect the ``gitaggregate`` command.
(See : https://pypi.org/project/git-aggregator/)
Repo files are pre-generated. You can update them with your custom settings.
(custom branches, extra PRs, git shallow options, etc...)
- ``extra_python_requirements.txt`` enumerates the list of extra python librairies
required to run the odoo instance.
The syntax should respect the ``pip install -r`` command.
(See : https://pip.pypa.io/en/stable/reference/requirements-file-format/)
- ``extra_debian_requirements.txt`` enumerates the list of extra system librairies
required to run the odoo instance.
- ``odoo.conf`` file. Add here extra configuration required for your custom modules.
the classical keys (``db_host``, ``db_port``, etc...) are automatically
autogenerated.
At this step, you should change the autogenerated files.
You can use default files, if you have a very simple odoo instance without custom code,
extra repositories, or dependencies...
**Note:**
- In your repos.yml, preserve ``openupgrade`` and ``server-tools`` repositories
to have all the features of the librairies available.
- In your repos.yml file, the odoo project should be in ``./src/odoo``
and the openupgrade project should be in ``./src/openupgrade/`` folder.
<a name="command-pull-submodule"/>
## Command: ``pull-submodule``
**Prerequites:** init + being in a git repository. (if not, you can simply run ``git init``)
if you already have a repos.yml file on github / gitlab, it can be convenient to
synchronize the repository, instead of copy past the ``repos.yml`` manually.
In that case, you can add extra values, in the ``config.yml`` file in the section
```yaml
odoo_version_settings:
12.0:
repo_url: url_of_the_repo_that_contains_a_repos_yml_file
repo_branch: 12.0
repo_file_path: repos.yml
```
then run following command :
```shell
odoo-openupgrade-wizard pull-submodule
```
<a name="command-get-code"/>
## Command: ``get-code``
**Prerequites:** init
```shell
odoo-openupgrade-wizard get-code
```
This command will simply get all the Odoo code required to run all the steps
for your migration with the ``gitaggregate`` tools.
The code is defined in the ``repos.yml`` of each environment folders. (or in the
directory ``repo_submodule`` if you use ``pull-submodule`` feature.)
**Note**
* This step could take a big while !
**Optional arguments**
if you want to update the code of some given versions, you can provide an extra parameter:
```shell
odoo-openupgrade-wizard get-code --versions 10.0,11.0
```
<a name="command-docker-build"/>
## Command: ``docker-build``
**Prerequites:** init + get-code
This will build local docker images that will be used in the following steps.
At this end of this step executing the following command should show a docker image per version.
```shell
docker images --filter "reference=odoo-openupgrade-wizard-*"
```
```
REPOSITORY TAG IMAGE ID CREATED SIZE
odoo-openupgrade-wizard-image---my-customer-10-12---12.0 latest ef664c366208 2 weeks ago 1.39GB
odoo-openupgrade-wizard-image---my-customer-10-12---11.0 latest 24e283fe4ae4 2 weeks ago 1.16GB
odoo-openupgrade-wizard-image---my-customer-10-12---10.0 latest 9d94dce2bd4e 2 weeks ago 924MB
```
**Optional arguments**
* if you want to (re)build an image for some given versions, you can provide
an extra parameter: ``--versions 10.0,12.0``
**Note**
* This step could take a big while also !
<a name="command-run"/>
## Command: ``run``
**Prerequites:** init + get-code + build
```shell
odoo-openupgrade-wizard run\
--step 1\
--database DB_NAME
```
Run an Odoo instance with the environment defined by the step argument.
The database will be created, if it doesn't exists.
if ``stop-after-init`` is disabled, the odoo instance will be available
at your host, at the following url : http://localhost:9069
(Port depends on your ``host_odoo_xmlrpc_port`` setting of your ``config.yml`` file)
**Optional arguments**
* You can add ``--init-modules=purchase,sale`` to install modules.
* You can add ``stop-after-init`` flag to turn off the process at the end
of the installation.
<a name="command-install-from-csv"/>
## Command: ``install-from-csv``
**Prerequites:** init + get-code + build
```shell
odoo-openupgrade-wizard install-from-csv\
--database DB_NAME
```
Install the list of the modules defined in your ``modules.csv`` files on the
given database.
The database will be created, if it doesn't exists.
To get a correct ``modules.csv`` file, the following query can be used:
```shell
psql -c "copy (select name, shortdesc from ir_module_module where state = 'installed' order by 1) to stdout csv" coopiteasy
```
<a name="command-upgrade"/>
## Command: ``upgrade``
**Prerequites:** init + get-code + build
```shell
odoo-openupgrade-wizard upgrade\
--database DB_NAME
```
Realize an upgrade of the database from the initial version to
the final version, following the different steps.
For each step, it will :
1. Execute the ``pre-migration.sql`` of the step.
2. Realize an "update all" (in an upgrade or update context)
3. Execute the scripts via XML-RPC (via ``odoorpc``) defined in
the ``post-migration.py`` file.
**Optional arguments**
* You can add ``--first-step=2`` to start at the second step.
* You can add ``--last-step=3`` to end at the third step.
<a name="command-generate-module-analysis"/>
## Command: ``generate-module-analysis``
**Prerequites:** init + get-code + build
```shell
odoo-openupgrade-wizard generate-module-analysis\
--database DB_NAME
--step 2
--modules MODULE_LIST
```
Realize an analyze between the target version (in parameter via the step argument)
and the previous version. It will generate analysis_file.txt files present
in OpenUpgrade project.
You can also use this fonction to analyze differences for custom / OCA modules
between several versions, in case of refactoring.
<a name="command-estimate-workload"/>
## Command: ``estimate-workload``
**Prerequites:** init + get-code
```shell
odoo-openupgrade-wizard estimate-workload
```
Generate an HTML file name ``analysis.html`` with all the information regarding
the work to do for the migration.
- checks that the modules are present in each version. (by managing the
renaming or merging of modules)
- check that the analysis and migration have been done for the official
modules present in odoo/odoo
<a name="command-psql"/>
## Command: ``psql``
**Prerequites:** init
```shell
odoo-openupgrade-wizard psql
--database DB_NAME
--command "SQL_REQUEST"
```
Execute an SQL Request on the target database.
**Optional arguments**
* If no ``database`` is provided, default ``postgres`` database will be used. exemple:
```shell
odoo-openupgrade-wizard psql --command "\l";
```
Result:
```
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
------------+-------+----------+------------+------------+-------------------
postgres | odoo | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | odoo | UTF8 | en_US.utf8 | en_US.utf8 | =c/odoo +
| | | | | odoo=CTc/odoo
template1 | odoo | UTF8 | en_US.utf8 | en_US.utf8 | =c/odoo +
| | | | | odoo=CTc/odoo
test_psql | odoo | UTF8 | en_US.utf8 | en_US.utf8 |
```
* if you execute request that return long result, you can choose to select ``pager`` or ``-no-pager``
option to display the result via the click function ``echo_via_pager``.
(see : https://click.palletsprojects.com/en/8.1.x/utils/#pager-support)
Note : Pager is enabled by default.
* you can pass extra psql arguments inline.
```shell
odoo-openupgrade-wizard psql
--database=test_psql
--command "select id, name from res_partner where name ilike '%admin%';"
-H
```
Result:
```html
<table border="1">
<tr>
<th align="center">id</th>
<th align="center">name</th>
</tr>
<tr valign="top">
<td align="right">3</td>
<td align="left">Administrator</td>
</tr>
</table>
<p>(1 row)<br />
</p>
```
See all the options here https://www.postgresql.org/docs/current/app-psql.html
<a name="command-copydb"/>
## Command: ``copydb``
**Prerequites:** init
```shell
odoo-openupgrade-wizard copydb
--source DB_NAME
--dest NEW_DB_NAME
```
Create an Odoo database by copying an existing one.
This script copies using postgres CREATEDB WITH TEMPLATE. It also copies
the filestore.
<a name="command-dropdb"/>
## Command: ``dropdb``
**Prerequites:** init
```shell
odoo-openupgrade-wizard dropdb
--database DB_NAME
```
Delete an Odoo database and its filestore.
This command will always success even if DB_NAME does not exists.
<a name="command-dumpdb"/>
## Command: ``dumpdb``
**Prerequites:** init
```shell
odoo-openupgrade-wizard dumpdb
--database DB_NAME
--database-path DATABASE_PATH
--filestore-path FILESTORE_PATH
```
Dump the database DB_NAME to DATABASE_PATH and export the filestore
related to DB_NAME into FILESTORE_PATH. To choose the format of the
backup files look at the `--database-format` and `--filestore-format`.
*WARNING*: DATABASE_PATH should be a sub directory of the project path
in orter to have the postgresql container able to write the dump file.
For example, the project path is `/path/to/myproject` (where you run the
`init` command), then DATABASE_PATH can be any of the subdirectory of
`/path/to/myproject`.
**Optional arguments**
* To chose the database format use `--database-format`. Format can be
one of the following:
- `p` for plain sql text
- `c` for custom compressed backup of `pg_dump`
- `d` for directory structure
- `t` for a tar version of the directory structure
See also https://www.postgresql.org/docs/current/app-pgdump.html
The default database format is `c`.
* To chose the filestore format use `--filestore-format`. Format can be
one of the following:
- `d` copy of the directory structure
- `t` tar version of the directory structure (not compressed)
- `tgz` tar version of the directory structure compressed with gzip.
The default filestore format is `tgz`.
* By default, if database file or filestore file already exists, the
command will fail, preserving the existing dump. If you need to
overwrite the existing files, the `--force` option can be used.

45
ROADMAP.md
View File

@ -1,45 +0,0 @@
# Python Version
* py310 is not available, due to dependencies to ``odoorpc`` that raise an error :
``ERROR tests/cli_A_init_test.py - AttributeError: module 'collections' has no attribute 'MutableMapping'``
Follow bug : https://stackoverflow.com/questions/69512672/getting-attributeerror-module-collections-has-no-attribute-mutablemapping-w
# openUpgradelib Versions
* ``openupgradelib`` requires a new feature psycopg2.sql since
(21 Aug 2019)
https://github.com/OCA/openupgradelib/commit/7408580e4469ba4b0cabb923da7facd71567a2fb
so we pin openupgradelib==2.0.0 (21 Jul 2018)
The python version in the Odoo:12 docker image is : ``Python 3.5.3 (default, Apr 5 2021, 09:00:41)`` that is very old.
- https://github.com/OCA/openupgradelib/issues/248
- https://github.com/OCA/openupgradelib/issues/288
- https://github.com/OCA/openupgradelib.git@ed01555b8ae20f66b3af178c8ecaf6edd110ce75#egg=openupgradelib
TODO : Fix via another way (other way than pining ``openuppgradelib`` version) the problem of old odoo versions. (it makes the upgrade failing for old revision (V8, etc...))
# Gitlab-CI
* for the time being, Features requiring ``odoorpc`` are failing in gitlab-CI.
Tests are working locally but there is a network problem. For that reason, tests witch names
begins by ``cli_2`` like (``cli_20_install_from_csv_test.py``) are disabled in ``.gitlab-ci.yml``.
TODO : work with Pierrick Brun, to run gitlab-runner on Akretion CI (without docker), to see if it is
fixing the problem.
# Features Work In Progress
* Add a tools to analyze workload.
# Possible Improvments
* select ``without-demo all`` depending on if the database
is created or not (and if current database contains demo data, checking if base.user_demo exists ?)
# Other points not in the scope of GRAP work
* Allow to use custom docker images.
* Check if there are default values for containers, limiting ressources.