hobbyhimmel/odoo-openupgrade-wizard
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[WIP]

Browse Source
This commit is contained in:
Sylvain LE GAL 2022-04-27 15:18:18 +02:00
parent 5a75cb81d2
commit dce91df36d
18 changed files with 349 additions and 141 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
DEVELOP.md
View File

@ -1,7 +1,7 @@
# Extra Developper Requirements
If you want to contribute to this library without installing anything in your
system,
If you want to use this library without installing anything in your
system, execute the following steps, otherwise, go to 'Installation' part.
1. Run a docker container :
@ -17,7 +17,7 @@ apt-get install git python3 python3-pip python3-venv
python3 -m pip install --user pipx
python3 -m pipx ensurepath
# re-login via su root
su root
pipx install virtualenv
pipx install poetry
@ -56,6 +56,15 @@ tox
Note : you should have all the python versions available in your local system.
```
sudo apt-get install python3.6 python3.6-distutils
sudo apt-get install python3.7 python3.7-distutils
sudo apt-get install python3.8 python3.8-distutils
sudo apt-get install python3.9 python3.9-distutils
sudo apt-get install python3.10 python3.10-distutils
```
## Via Gitlab Runner locally

122
README.md
View File

@ -11,14 +11,21 @@ this tool is useful for complex migrations:
It will create a migration environment (with all the code available)
and provides helpers to run (and replay) migrations until it works.
## Commands
# Installation
### ``odoo-openupgrade-wizard init``
``pipx install odoo-openupgrade-wizard``.
To develop and contribute to the library, refer to the ``DEVELOP.md`` file.
# Usage
## ``odoo-openupgrade-wizard init``
```
odoo-openupgrade-wizard init\
--initial-version=10.0\
--final-version=12.0\
--initial-release=10.0\
--final-release=12.0\
--project-name=my-customer-10-12\
--extra-repository=OCA/web,OCA/server-tools
```
@ -30,14 +37,6 @@ config.yml
log/
2022_03_25__23_12_41__init.log
...
repos/
10.0.yml
11.0.yml
12.0.yml
requirements/
10.0_requirements.txt
11.0_requirements.txt
12.0_requirements.txt
scripts/
step_1__update__10.0/
pre-migration.sql
@ -52,31 +51,94 @@ scripts/
pre-migration.sql
post-migration.py
src/
env_10.0/
debian_requirements.txt
Dockerfile
odoo.cfg
python_requirements.txt
repos.yml
src/
odoo/
openupgrade/
env_11.0/
...
env_12.0/
...
```
* ``log/`` will contains all the log of the ``odoo-openupgrade-wizard``
* ``log`` folder will contains all the log of the ``odoo-openupgrade-wizard``
and the logs of the odoo instance that will be executed.
* ``repos/`` contains a file per version of odoo, that enumerates the
list of the repositories to use to run each odoo instance.
The syntax should respect the ``gitaggregate`` command.
(See : https://pypi.org/project/git-aggregator/)
Repo files are pre-generated. You can update them with your custom settings.
(custom branches, extra PRs, git shallow options, etc...)
* ``requirements/`` contains a file per version of odoo, that enumerates the
list of extra python librairies required to run each odoo instance.
The syntax should respect the ``pip install -r`` command.
(See : https://pip.pypa.io/en/stable/reference/requirements-file-format/)
* ``scripts`` contains a folder per migration step. In each step folder:
* ``scripts`` folder contains a folder per migration step. In each step folder:
- ``pre-migration.sql`` can contains extra SQL queries you want to execute
before beginning the step.
- ``post-migration.py`` can contains extra python command to execute
after the execution of the step. (the orm will be available)
# TODO
* ``src`` folder contains a folder per Odoo version. In each environment folder:
* with coop it easy :
- short_help of group decorator ? seems useless...
- ``repos.yml`` enumerates the list of the repositories to use to run the odoo instance.
The syntax should respect the ``gitaggregate`` command.
(See : https://pypi.org/project/git-aggregator/)
Repo files are pre-generated. You can update them with your custom settings.
(custom branches, extra PRs, git shallow options, etc...)
- ``python_requirements.txt`` enumerates the list of extra python librairies
required to run the odoo instance.
The syntax should respect the ``pip install -r`` command.
(See : https://pip.pypa.io/en/stable/reference/requirements-file-format/)
- ``debian_requirements.txt`` enumerates the list of extra system librairies
required to run the odoo instance.
At this step, you should change the autogenerated files.
You can use default files, if you have a very simple odoo instance without custom code,
extra repositories, or dependencies...
## ``odoo-openupgrade-wizard get-code``
```
odoo-openupgrade-wizard get-code
```
This command will simply get all the Odoo code required to run all the steps for your migration.
The code is defined in the ``repos.yml`` of each sub folders.
Note : This step could take a big while !
**Optional arguments**
if you want to update the code of some given releases, you can provide an extra parameter:
```
odoo-openupgrade-wizard get-code --releases 10.0,11.0
```
## ``odoo-openupgrade-wizard docker-build``
This will build local docker images that will be used in the following steps.
This script will pull official odoo docker images, defined in the ``Dockerfile`` of
each folder, and build a custom images on top the official one, installing inside
custom librairies defined in ``debian_requirements.txt``, ``python_requirements.txt``.
At this end of this step executing the following command
``docker images --filter "reference=odoo-openupgrade-wizard-*"`` should show a docker image per version.
```
REPOSITORY TAG IMAGE ID CREATED SIZE
odoo-openupgrade-wizard-image---my-customer-10-12---12.0 latest ef664c366208 2 weeks ago 1.39GB
odoo-openupgrade-wizard-image---my-customer-10-12---11.0 latest 24e283fe4ae4 2 weeks ago 1.16GB
odoo-openupgrade-wizard-image---my-customer-10-12---10.0 latest 9d94dce2bd4e 2 weeks ago 924MB
```
**Optional arguments**
if you want to build an image for some given releases, you can provide an extra parameter:
```
odoo-openupgrade-wizard docker-build --releases 10.0,12.0
```

6
ROADMAP.md Normal file
View File

@ -0,0 +1,6 @@
# TODO
* with coop it easy :
- short_help of group decorator ? seems useless...
* add constrains on ``--step`` option.

2
odoo_openupgrade_wizard/cli.py
View File

@ -59,7 +59,7 @@ def main(ctx, env_folder, filestore_folder):
filestore_folder_path = Path(filestore_folder)
# ensure log folder exists
ensure_folder_exists(log_folder_path)
ensure_folder_exists(log_folder_path, git_ignore_content=True)
# Create log file
log_file_path = log_folder_path / Path(

11
odoo_openupgrade_wizard/cli_init.py
View File

@ -96,7 +96,7 @@ def init(
count += 1
# add final update step
count += 1
if len(odoo_versions) > 1:
steps.append(
{
"name": count,
@ -111,7 +111,9 @@ def init(
ensure_folder_exists(ctx.obj["src_folder_path"])
# 4. ensure filestore folder exists
ensure_folder_exists(ctx.obj["filestore_folder_path"])
ensure_folder_exists(
ctx.obj["filestore_folder_path"], git_ignore_content=True
)
# 5. ensure main configuration file exists
ensure_file_exists_from_template(
@ -162,6 +164,11 @@ def init(
odoo_version=odoo_version,
)
# Create 'src' folder that will contain all the odoo code
ensure_folder_exists(
path_version / Path("src"), git_ignore_content=True
)
# 6. Create one folder per step and add files
ensure_folder_exists(ctx.obj["script_folder_path"])

28
odoo_openupgrade_wizard/cli_run.py
View File

@ -1,5 +1,3 @@
from time import sleep
import click
from loguru import logger
@ -15,22 +13,32 @@ from odoo_openupgrade_wizard.tools_odoo import kill_odoo, run_odoo
@step_option
@database_option
@click.option(
"--duration",
type=float,
"--stop-after-init",
is_flag=True,
default=False,
help="Duration of the execution of the script. Mainly used"
help="Stop after init. Mainly used"
" for test purpose, for commands that are using input()"
" function to stop.",
)
@click.option(
"--init-modules",
type=str,
help="List of modules to install. Equivalent to -i odoo options.",
)
@click.pass_context
def run(ctx, step, database, duration):
def run(ctx, step, database, stop_after_init, init_modules):
migration_step = get_migration_step_from_options(ctx, step)
try:
run_odoo(ctx, migration_step, database=database)
if duration:
sleep(duration)
else:
run_odoo(
ctx,
migration_step,
database=database,
detached_container=not stop_after_init,
init=init_modules,
stop_after_init=stop_after_init,
)
if not stop_after_init:
input("Press 'Enter' to kill the odoo container and exit ...")
except (KeyboardInterrupt, SystemExit):
logger.info("Received Keyboard Interrupt or System Exiting...")

10
odoo_openupgrade_wizard/templates.py
View File

@ -1,5 +1,7 @@
CONFIG_YML_TEMPLATE = """project_name: {{ project_name }}
host_odoo_xmlrpc_port: 8069
odoo_versions:
{% for odoo_version in odoo_versions %}
- release: {{ odoo_version['release'] }}
@ -96,6 +98,9 @@ RUN apt-get update || true &&\
RUN {{ odoo_version["python_major_version"] }}\
-m pip install -r python_requirements.txt
# VOLUME ["/var/lib/odoo2"]
# Reset to odoo user to run the container
USER odoo
"""
@ -106,3 +111,8 @@ POST_MIGRATION_PY_TEMPLATE = """
def main(self, step):
pass
"""
GIT_IGNORE_CONTENT = """
*
!.gitignore
"""

74
odoo_openupgrade_wizard/tools_docker.py Normal file
View File

@ -0,0 +1,74 @@
import docker
from loguru import logger
def run_container(
image_name,
container_name,
command=False,
ports=False,
volumes=False,
links=False,
detach=False,
auto_remove=False,
):
client = docker.from_env()
logger.info("Launching Docker container named %s ..." % (image_name))
debug_docker_command = "docker run --name %s\\\n" % (container_name)
if ports:
for internal_port, host_port in ports.items():
debug_docker_command += (
" --publish {host_port}:{internal_port}\\\n".format(
internal_port=internal_port, host_port=host_port
)
)
if volumes:
for volume in volumes:
external_path, internal_path = volume.split(":")
debug_docker_command += (
" --volume {external_path}:{internal_path}\\\n".format(
external_path=external_path, internal_path=internal_path
)
)
if links:
for k, v in links.items():
debug_docker_command += " --link {k}:{v}\\\n".format(k=k, v=v)
if auto_remove:
debug_docker_command += " --rm"
if detach:
debug_docker_command += " --detach"
debug_docker_command += " %s\\\n" % (image_name)
debug_docker_command += " %s" % (command)
logger.debug(debug_docker_command)
container = client.containers.run(
image_name,
name=container_name,
command=command,
ports=ports,
volumes=volumes,
links=links,
detach=detach,
auto_remove=auto_remove,
)
if detach:
logger.info("Container launched.")
elif auto_remove:
logger.info("Container closed.")
return container
def kill_container(container_name):
client = docker.from_env()
containers = client.containers.list(
all=True,
filters={"name": container_name},
)
for container in containers:
logger.info(
"Stop container %s, based on image '%s'."
% (container.name, ",".join(container.image.tags))
)
container.stop()

80
odoo_openupgrade_wizard/tools_odoo.py
View File

@ -1,7 +1,6 @@
from pathlib import Path
import docker
from loguru import logger
from odoo_openupgrade_wizard.tools_docker import kill_container, run_container
# WIP
@ -42,7 +41,7 @@ def get_odoo_env_path(ctx, odoo_version: dict) -> Path:
def get_docker_image_tag(ctx, odoo_version: dict) -> str:
"""Return a docker image tag, based on project name and odoo release"""
return "odoo-openupgrade-wizard-image-%s-%s" % (
return "odoo-openupgrade-wizard-image__%s__%s" % (
ctx.obj["config"]["project_name"],
str(odoo_version["release"]).rjust(4, "0"),
)
@ -51,7 +50,7 @@ def get_docker_image_tag(ctx, odoo_version: dict) -> str:
def get_docker_container_name(ctx, migration_step: dict) -> str:
"""Return a docker container name, based on project name,
odoo release and migration step"""
return "odoo-openupgrade-wizard-container-%s-%s-step-%s" % (
return "odoo-openupgrade-wizard-container---%s---%s---step---%s" % (
ctx.obj["config"]["project_name"],
str(migration_step["release"]).rjust(4, "0"),
str(migration_step["name"]).rjust(2, "0"),
@ -70,83 +69,84 @@ def generate_odoo_command(
ctx,
migration_step: dict,
database: str,
update_all: bool,
update: str,
init: str,
stop_after_init: bool,
shell: bool,
demo: bool,
) -> str:
# TODO, make it dynamic
addons_path = (
"/container_env/src/odoo/addons," "/container_env/src/odoo/odoo/addons"
)
addons_path = "/odoo_env/src/odoo/addons," "/odoo_env/src/odoo/odoo/addons"
database_cmd = database and "--database %s" % database or ""
update_all_cmd = update_all and "--update_all" or ""
update_cmd = update and "--update_%s" % update or ""
init_cmd = init and "--init %s" % init or ""
stop_after_init_cmd = stop_after_init and "--stop-after-init" or ""
shell_cmd = shell and "shell" or ""
return (
f"/container_env/src/odoo/odoo-bin"
demo_cmd = not demo and "--without-demo all" or ""
# TODO, FIXME
# log_file = "/env/log/%s" % "log.log"
result = (
f"/odoo_env/src/odoo/odoo-bin"
f" --db_host db"
f" --db_port 5432"
f" --db_user odoo"
f" --db_password odoo"
f" --workers 0"
f" --config /odoo_env/odoo.cfg"
# f" --data-dir /env/filestore/"
# f" --logfile {log_file}"
f" --addons-path {addons_path}"
f" {database_cmd}"
f" {update_all_cmd}"
f" {update_cmd}"
f" {init_cmd}"
f" {stop_after_init_cmd}"
f" {shell_cmd}"
f" {demo_cmd}"
)
return result
def run_odoo(
ctx,
migration_step: dict,
detached_container: bool = False,
database: str = False,
update: str = False,
init: str = False,
stop_after_init: bool = False,
shell: bool = False,
update_all: bool = False,
demo: bool = False,
):
client = docker.from_env()
# TODO, check if stop_after_init and detached_container are redondant.
odoo_version = get_odoo_version_from_migration_step(ctx, migration_step)
folder_path = get_odoo_env_path(ctx, odoo_version)
env_path = ctx.obj["env_folder_path"]
odoo_env_path = get_odoo_env_path(ctx, odoo_version)
command = generate_odoo_command(
ctx,
migration_step,
database=database,
update=update,
init=init,
stop_after_init=stop_after_init,
shell=shell,
update_all=update_all,
demo=demo,
)
image_name = get_docker_image_tag(ctx, odoo_version)
container_name = get_docker_container_name(ctx, migration_step)
logger.info(
"Launching Odoo Docker container named %s based on image '%s'."
% (container_name, image_name)
)
container = client.containers.run(
image_name,
name=container_name,
return run_container(
get_docker_image_tag(ctx, odoo_version),
get_docker_container_name(ctx, migration_step),
command=command,
ports={"8069": 8069, "5432": 5432},
volumes=["%s:/container_env/" % (folder_path)],
volumes=[
"%s:/env/" % (env_path),
"%s:/odoo_env/" % (odoo_env_path),
],
links={"db": "db"},
detach=True,
detach=detached_container,
auto_remove=True,
)
logger.info("Container Launched. Command executed : %s" % command)
return container
def kill_odoo(ctx, migration_step: dict):
client = docker.from_env()
containers = client.containers.list(
all=True,
filters={"name": get_docker_container_name(ctx, migration_step)},
)
for container in containers:
logger.info(
"Stop container %s, based on image '%s'."
% (container.name, ",".join(container.image.tags))
)
container.stop()
kill_container(get_docker_container_name(ctx, migration_step))

12
odoo_openupgrade_wizard/tools_system.py
View File

@ -8,8 +8,12 @@ from jinja2 import Template
from loguru import logger
from plumbum.cmd import mkdir
from odoo_openupgrade_wizard import templates
def ensure_folder_exists(folder_path: Path, mode: str = False):
def ensure_folder_exists(
folder_path: Path, mode: str = False, git_ignore_content: bool = False
):
"""Create a local folder.
- directory is created if it doesn't exist.
- mode is applied if defined.
@ -22,6 +26,12 @@ def ensure_folder_exists(folder_path: Path, mode: str = False):
logger.info("Creating folder '%s' ..." % (folder_path))
mkdir(cmd)
if git_ignore_content:
ensure_file_exists_from_template(
folder_path / Path(".gitignore"),
templates.GIT_IGNORE_CONTENT,
)
def ensure_file_exists_from_template(
file_path: Path, template_name: str, **args

6
pyproject.toml
View File

@ -17,7 +17,11 @@ classifiers = [
"Intended Audience :: Developers",
"Development Status :: 2 - Pre-Alpha",
"Operating System :: Unix",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Framework :: Odoo",
]

23
tests/__init__.py Normal file
View File

@ -0,0 +1,23 @@
import logging
from click.testing import CliRunner
from odoo_openupgrade_wizard.cli import main
_logger = logging.getLogger()
def assert_result_cli_invoke(result):
pass
def cli_runner_invoke(cmd):
result = CliRunner().invoke(
main,
cmd,
catch_exceptions=False,
)
if not result.exit_code == 0:
_logger.error("exit_code: %s" % result.exit_code)
_logger.error("output: %s" % result.output)
assert result.exit_code == 0

11
tests/cli_A_init_test.py
View File

@ -1,18 +1,17 @@
import filecmp
from pathlib import Path
from click.testing import CliRunner
from plumbum.cmd import mkdir
from odoo_openupgrade_wizard.cli import main
from . import cli_runner_invoke
def test_cli_init():
output_folder_path = Path("./tests/output_A")
expected_folder_path = Path("./tests/output_A_expected")
mkdir([output_folder_path, "--parents"])
result = CliRunner().invoke(
main,
cli_runner_invoke(
[
"--env-folder=%s" % output_folder_path,
"init",
@ -21,10 +20,8 @@ def test_cli_init():
"--final-release=12.0",
"--extra-repository="
"OCA/web,OCA/server-tools,GRAP/grap-odoo-incubator",
],
catch_exceptions=False,
]
)
assert result.exit_code == 0
assert filecmp.cmp(
output_folder_path / Path("config.yml"),

16
tests/cli_B_01_get_code_test.py
View File

@ -1,9 +1,8 @@
from pathlib import Path
from click.testing import CliRunner
from plumbum.cmd import mkdir
from odoo_openupgrade_wizard.cli import main
from . import cli_runner_invoke
def test_cli_get_code():
@ -12,8 +11,7 @@ def test_cli_get_code():
# We initialize an env with only one version to avoid to git clone
# large data
CliRunner().invoke(
main,
cli_runner_invoke(
[
"--env-folder=%s" % output_folder_path,
"init",
@ -21,19 +19,15 @@ def test_cli_get_code():
"--initial-release=14.0",
"--final-release=14.0",
"--extra-repository=OCA/web",
],
catch_exceptions=False,
]
)
result = CliRunner().invoke(
main,
cli_runner_invoke(
[
"--env-folder=%s" % output_folder_path,
"get-code",
],
catch_exceptions=False,
]
)
assert result.exit_code == 0
openupgrade_path = output_folder_path / Path(
"./src/env_14.0/src/openupgrade"

12
tests/cli_B_02_docker_build_test.py
View File

@ -1,27 +1,23 @@
from pathlib import Path
import docker
from click.testing import CliRunner
from odoo_openupgrade_wizard.cli import main
from . import cli_runner_invoke
def test_cli_docker_build():
output_folder_path = Path("./tests/output_B")
result = CliRunner().invoke(
main,
cli_runner_invoke(
[
"--env-folder=%s" % output_folder_path,
"docker-build",
"--releases=14.0",
],
catch_exceptions=False,
]
)
assert result.exit_code == 0
docker_client = docker.from_env()
assert docker_client.images.get(
"odoo-openupgrade-wizard-image-test-cli-14.0"
"odoo-openupgrade-wizard-image__test-cli__14.0"
)

34
tests/cli_B_03_run_test.py
View File

@ -1,28 +1,34 @@
from pathlib import Path
import docker
from click.testing import CliRunner
from . import cli_runner_invoke
from odoo_openupgrade_wizard.cli import main
# import docker
def test_cli_run():
output_folder_path = Path("./tests/output_B")
result = CliRunner().invoke(
main,
db_name = "database_test_cli_run_3"
cli_runner_invoke(
[
"--env-folder=%s" % output_folder_path,
"run",
"--step=1",
"--duration=2",
],
catch_exceptions=False,
"--database=%s" % db_name,
"--init-modules=base",
"--stop-after-init",
]
)
assert result.exit_code == 0
# Ensure that all the containers are removed
docker_client = docker.from_env()
assert not docker_client.containers.list(
all=True, filters={"name": "odoo-openupgrade-wizard"}
)
# # Ensure that a subfolder filestore/DB_NAME has been created
# db_filestore_path = output_folder_path / Path(
# "./filestore/filestore/%s" % db_name
# )
# assert db_filestore_path.exists()
# # Ensure that all the containers are removed
# docker_client = docker.from_env()
# assert not docker_client.containers.list(
# all=True, filters={"name": "odoo-openupgrade-wizard"}
# )

6
tests/output_A_expected/config.yml
View File

@ -1,5 +1,7 @@
project_name: test-cli
host_odoo_xmlrpc_port: 8069
odoo_versions:
- release: 9.0
@ -33,7 +35,7 @@ migration_steps:
action: upgrade
complete_name: step_04__upgrade__12.0
- name: 6
- name: 5
release: 12.0
action: update
complete_name: step_06__update__12.0
complete_name: step_05__update__12.0

2
tox.ini
View File

@ -1,7 +1,7 @@
[tox]
isolated_build = true
skipsdist = True
envlist = py36, py37, py38
envlist = py36, py37, py38, py39, py310
[testenv]
whitelist_externals = poetry