[WIP]

DEVELOP.md

@@ -1,7 +1,7 @@
 # Extra Developper Requirements
 
-If you want to contribute to this library without installing anything in your
+If you want to use this library without installing anything in your
-system,
+system, execute the following steps, otherwise, go to 'Installation' part.
 
 1. Run a docker container :
 
@@ -17,7 +17,7 @@ apt-get install git python3 python3-pip python3-venv
 python3 -m pip install --user pipx
 python3 -m pipx ensurepath
 
-# re-login via su root
+su root
 
 pipx install virtualenv
 pipx install poetry
@@ -56,6 +56,15 @@ tox
 
 Note : you should have all the python versions available in your local system.
 
+
+```
+sudo apt-get install python3.6  python3.6-distutils
+sudo apt-get install python3.7  python3.7-distutils
+sudo apt-get install python3.8  python3.8-distutils
+sudo apt-get install python3.9  python3.9-distutils
+sudo apt-get install python3.10 python3.10-distutils
+```
+
 ## Via Gitlab Runner locally
 
 

README.md

@@ -11,14 +11,21 @@ this tool is useful for complex migrations:
 It will create a migration environment (with all the code available)
 and provides helpers to run (and replay) migrations until it works.
 
-## Commands
+# Installation
 
-### ``odoo-openupgrade-wizard init``
+``pipx install odoo-openupgrade-wizard``.
+
+To develop and contribute to the library, refer to the ``DEVELOP.md`` file.
+
+# Usage
+
+## ``odoo-openupgrade-wizard init``
 
 ```
 odoo-openupgrade-wizard init\
-  --initial-version=10.0\
+  --initial-release=10.0\
-  --final-version=12.0\
+  --final-release=12.0\
+  --project-name=my-customer-10-12\
   --extra-repository=OCA/web,OCA/server-tools
 ```
 
@@ -30,14 +37,6 @@ config.yml
 log/
     2022_03_25__23_12_41__init.log
     ...
-repos/
-    10.0.yml
-    11.0.yml
-    12.0.yml
-requirements/
-    10.0_requirements.txt
-    11.0_requirements.txt
-    12.0_requirements.txt
 scripts/
     step_1__update__10.0/
         pre-migration.sql
@@ -52,31 +51,94 @@ scripts/
         pre-migration.sql
         post-migration.py
 src/
+    env_10.0/
+        debian_requirements.txt
+        Dockerfile
+        odoo.cfg
+        python_requirements.txt
+        repos.yml
+        src/
+            odoo/
+            openupgrade/
+    env_11.0/
+        ...
+    env_12.0/
+        ...
+
 ```
 
-* ``log/`` will contains all the log of the ``odoo-openupgrade-wizard``
+* ``log`` folder will contains all the log of the ``odoo-openupgrade-wizard``
   and the logs of the odoo instance that will be executed.
 
-
+* ``scripts`` folder contains a folder per migration step. In each step folder:
-* ``repos/`` contains a file per version of odoo, that enumerates the
-  list of the repositories to use to run each odoo instance.
-  The syntax should respect the ``gitaggregate`` command.
-  (See : https://pypi.org/project/git-aggregator/)
-  Repo files are pre-generated. You can update them with your custom settings.
-  (custom branches, extra PRs, git shallow options, etc...)
-
-* ``requirements/`` contains a file per version of odoo, that enumerates the
-  list of extra python librairies required to run each odoo instance.
-  The syntax should respect the ``pip install -r`` command.
-  (See : https://pip.pypa.io/en/stable/reference/requirements-file-format/)
-
-* ``scripts`` contains a folder per migration step. In each step folder:
   - ``pre-migration.sql`` can contains extra SQL queries you want to execute
     before beginning the step.
   - ``post-migration.py`` can contains extra python command to execute
     after the execution of the step. (the orm will be available)
 
-# TODO
+* ``src`` folder contains a folder per Odoo version. In each environment folder:
 
-* with coop it easy :
+    - ``repos.yml`` enumerates the list of the repositories to use to run the odoo instance.
-- short_help of group decorator ? seems useless...
+      The syntax should respect the ``gitaggregate`` command.
+      (See : https://pypi.org/project/git-aggregator/)
+      Repo files are pre-generated. You can update them with your custom settings.
+      (custom branches, extra PRs, git shallow options, etc...)
+
+    - ``python_requirements.txt`` enumerates the list of extra python librairies
+      required to run the odoo instance.
+      The syntax should respect the ``pip install -r`` command.
+      (See : https://pip.pypa.io/en/stable/reference/requirements-file-format/)
+
+    - ``debian_requirements.txt`` enumerates the list of extra system librairies
+      required to run the odoo instance.
+
+At this step, you should change the autogenerated files.
+You can use default files, if you have a very simple odoo instance without custom code,
+extra repositories, or dependencies...
+
+
+## ``odoo-openupgrade-wizard get-code``
+
+```
+odoo-openupgrade-wizard get-code
+```
+
+This command will simply get all the Odoo code required to run all the steps for your migration.
+The code is defined in the ``repos.yml`` of each sub folders.
+
+Note : This step could take a big while !
+
+**Optional arguments**
+
+if you want to update the code of some given releases, you can provide an extra parameter:
+
+```
+odoo-openupgrade-wizard get-code --releases 10.0,11.0
+```
+
+## ``odoo-openupgrade-wizard docker-build``
+
+This will build local docker images that will be used in the following steps.
+
+This script will pull official odoo docker images, defined in the ``Dockerfile`` of
+each folder, and build a custom images on top the official one, installing inside
+custom librairies defined in ``debian_requirements.txt``, ``python_requirements.txt``.
+
+At this end of this step executing the following command
+``docker images --filter "reference=odoo-openupgrade-wizard-*"`` should show a docker image per version.
+
+
+```
+REPOSITORY                                                 TAG       IMAGE ID       CREATED       SIZE
+odoo-openupgrade-wizard-image---my-customer-10-12---12.0   latest    ef664c366208   2 weeks ago   1.39GB
+odoo-openupgrade-wizard-image---my-customer-10-12---11.0   latest    24e283fe4ae4   2 weeks ago   1.16GB
+odoo-openupgrade-wizard-image---my-customer-10-12---10.0   latest    9d94dce2bd4e   2 weeks ago   924MB
+```
+
+**Optional arguments**
+
+if you want to build an image for some given releases, you can provide an extra parameter:
+
+```
+odoo-openupgrade-wizard docker-build --releases 10.0,12.0
+```

ROADMAP.md

@@ -0,0 +1,6 @@
+# TODO
+
+* with coop it easy :
+- short_help of group decorator ? seems useless...
+
+* add constrains on ``--step`` option.

odoo_openupgrade_wizard/cli.py

@@ -59,7 +59,7 @@ def main(ctx, env_folder, filestore_folder):
         filestore_folder_path = Path(filestore_folder)
 
     # ensure log folder exists
-    ensure_folder_exists(log_folder_path)
+    ensure_folder_exists(log_folder_path, git_ignore_content=True)
 
     # Create log file
     log_file_path = log_folder_path / Path(

odoo_openupgrade_wizard/cli_init.py

@@ -96,7 +96,7 @@ def init(
         count += 1
 
     # add final update step
-    count += 1
+    if len(odoo_versions) > 1:
         steps.append(
             {
                 "name": count,
@@ -111,7 +111,9 @@ def init(
     ensure_folder_exists(ctx.obj["src_folder_path"])
 
     # 4. ensure filestore folder exists
-    ensure_folder_exists(ctx.obj["filestore_folder_path"])
+    ensure_folder_exists(
+        ctx.obj["filestore_folder_path"], git_ignore_content=True
+    )
 
     # 5. ensure main configuration file exists
     ensure_file_exists_from_template(
@@ -162,6 +164,11 @@ def init(
             odoo_version=odoo_version,
         )
 
+        # Create 'src' folder that will contain all the odoo code
+        ensure_folder_exists(
+            path_version / Path("src"), git_ignore_content=True
+        )
+
     # 6. Create one folder per step and add files
     ensure_folder_exists(ctx.obj["script_folder_path"])
 

odoo_openupgrade_wizard/cli_run.py

@@ -1,5 +1,3 @@
-from time import sleep
-
 import click
 from loguru import logger
 
@@ -15,22 +13,32 @@ from odoo_openupgrade_wizard.tools_odoo import kill_odoo, run_odoo
 @step_option
 @database_option
 @click.option(
-    "--duration",
+    "--stop-after-init",
-    type=float,
+    is_flag=True,
     default=False,
-    help="Duration of the execution of the script. Mainly used"
+    help="Stop after init. Mainly used"
     " for test purpose, for commands that are using input()"
     " function to stop.",
 )
+@click.option(
+    "--init-modules",
+    type=str,
+    help="List of modules to install. Equivalent to -i odoo options.",
+)
 @click.pass_context
-def run(ctx, step, database, duration):
+def run(ctx, step, database, stop_after_init, init_modules):
 
     migration_step = get_migration_step_from_options(ctx, step)
     try:
-        run_odoo(ctx, migration_step, database=database)
+        run_odoo(
-        if duration:
+            ctx,
-            sleep(duration)
+            migration_step,
-        else:
+            database=database,
+            detached_container=not stop_after_init,
+            init=init_modules,
+            stop_after_init=stop_after_init,
+        )
+        if not stop_after_init:
             input("Press 'Enter' to kill the odoo container and exit ...")
     except (KeyboardInterrupt, SystemExit):
         logger.info("Received Keyboard Interrupt or System Exiting...")

odoo_openupgrade_wizard/templates.py

@@ -1,5 +1,7 @@
 CONFIG_YML_TEMPLATE = """project_name: {{ project_name }}
 
+host_odoo_xmlrpc_port: 8069
+
 odoo_versions:
 {% for odoo_version in odoo_versions %}
   - release: {{ odoo_version['release'] }}
@@ -96,6 +98,9 @@ RUN apt-get update || true &&\
 RUN {{ odoo_version["python_major_version"] }}\
  -m pip install -r python_requirements.txt
 
+
+# VOLUME ["/var/lib/odoo2"]
+
 # Reset to odoo user to run the container
 USER odoo
 """
@@ -106,3 +111,8 @@ POST_MIGRATION_PY_TEMPLATE = """
 def main(self, step):
     pass
 """
+
+GIT_IGNORE_CONTENT = """
+*
+!.gitignore
+"""

odoo_openupgrade_wizard/tools_docker.py

@@ -0,0 +1,74 @@
+import docker
+from loguru import logger
+
+
+def run_container(
+    image_name,
+    container_name,
+    command=False,
+    ports=False,
+    volumes=False,
+    links=False,
+    detach=False,
+    auto_remove=False,
+):
+    client = docker.from_env()
+
+    logger.info("Launching Docker container named %s ..." % (image_name))
+    debug_docker_command = "docker run --name %s\\\n" % (container_name)
+    if ports:
+        for internal_port, host_port in ports.items():
+            debug_docker_command += (
+                " --publish {host_port}:{internal_port}\\\n".format(
+                    internal_port=internal_port, host_port=host_port
+                )
+            )
+    if volumes:
+        for volume in volumes:
+            external_path, internal_path = volume.split(":")
+            debug_docker_command += (
+                " --volume {external_path}:{internal_path}\\\n".format(
+                    external_path=external_path, internal_path=internal_path
+                )
+            )
+    if links:
+        for k, v in links.items():
+            debug_docker_command += " --link {k}:{v}\\\n".format(k=k, v=v)
+    if auto_remove:
+        debug_docker_command += " --rm"
+    if detach:
+        debug_docker_command += " --detach"
+    debug_docker_command += " %s\\\n" % (image_name)
+    debug_docker_command += " %s" % (command)
+    logger.debug(debug_docker_command)
+
+    container = client.containers.run(
+        image_name,
+        name=container_name,
+        command=command,
+        ports=ports,
+        volumes=volumes,
+        links=links,
+        detach=detach,
+        auto_remove=auto_remove,
+    )
+    if detach:
+        logger.info("Container launched.")
+    elif auto_remove:
+        logger.info("Container closed.")
+
+    return container
+
+
+def kill_container(container_name):
+    client = docker.from_env()
+    containers = client.containers.list(
+        all=True,
+        filters={"name": container_name},
+    )
+    for container in containers:
+        logger.info(
+            "Stop container %s, based on image '%s'."
+            % (container.name, ",".join(container.image.tags))
+        )
+        container.stop()

odoo_openupgrade_wizard/tools_odoo.py

@@ -1,7 +1,6 @@
 from pathlib import Path
 
-import docker
+from odoo_openupgrade_wizard.tools_docker import kill_container, run_container
-from loguru import logger
 
 
 # WIP
@@ -42,7 +41,7 @@ def get_odoo_env_path(ctx, odoo_version: dict) -> Path:
 
 def get_docker_image_tag(ctx, odoo_version: dict) -> str:
     """Return a docker image tag, based on project name and odoo release"""
-    return "odoo-openupgrade-wizard-image-%s-%s" % (
+    return "odoo-openupgrade-wizard-image__%s__%s" % (
         ctx.obj["config"]["project_name"],
         str(odoo_version["release"]).rjust(4, "0"),
     )
@@ -51,7 +50,7 @@ def get_docker_image_tag(ctx, odoo_version: dict) -> str:
 def get_docker_container_name(ctx, migration_step: dict) -> str:
     """Return a docker container name, based on project name,
     odoo release and migration step"""
-    return "odoo-openupgrade-wizard-container-%s-%s-step-%s" % (
+    return "odoo-openupgrade-wizard-container---%s---%s---step---%s" % (
         ctx.obj["config"]["project_name"],
         str(migration_step["release"]).rjust(4, "0"),
         str(migration_step["name"]).rjust(2, "0"),
@@ -70,83 +69,84 @@ def generate_odoo_command(
     ctx,
     migration_step: dict,
     database: str,
-    update_all: bool,
+    update: str,
+    init: str,
     stop_after_init: bool,
     shell: bool,
+    demo: bool,
 ) -> str:
     # TODO, make it dynamic
-    addons_path = (
+    addons_path = "/odoo_env/src/odoo/addons," "/odoo_env/src/odoo/odoo/addons"
-        "/container_env/src/odoo/addons," "/container_env/src/odoo/odoo/addons"
-    )
     database_cmd = database and "--database %s" % database or ""
-    update_all_cmd = update_all and "--update_all" or ""
+    update_cmd = update and "--update_%s" % update or ""
-    stop_after_init_cmd = stop_after_init and "-- stop-after-init" or ""
+    init_cmd = init and "--init %s" % init or ""
+    stop_after_init_cmd = stop_after_init and "--stop-after-init" or ""
     shell_cmd = shell and "shell" or ""
-    return (
+    demo_cmd = not demo and "--without-demo all" or ""
-        f"/container_env/src/odoo/odoo-bin"
+    # TODO, FIXME
+    # log_file = "/env/log/%s" % "log.log"
+    result = (
+        f"/odoo_env/src/odoo/odoo-bin"
         f" --db_host db"
         f" --db_port 5432"
         f" --db_user odoo"
         f" --db_password odoo"
         f" --workers 0"
+        f" --config /odoo_env/odoo.cfg"
+        # f" --data-dir /env/filestore/"
+        # f" --logfile {log_file}"
         f" --addons-path {addons_path}"
         f" {database_cmd}"
-        f" {update_all_cmd}"
+        f" {update_cmd}"
+        f" {init_cmd}"
         f" {stop_after_init_cmd}"
         f" {shell_cmd}"
+        f" {demo_cmd}"
     )
+    return result
 
 
 def run_odoo(
     ctx,
     migration_step: dict,
+    detached_container: bool = False,
     database: str = False,
+    update: str = False,
+    init: str = False,
     stop_after_init: bool = False,
     shell: bool = False,
-    update_all: bool = False,
+    demo: bool = False,
 ):
-    client = docker.from_env()
+    # TODO, check if stop_after_init and detached_container are redondant.
     odoo_version = get_odoo_version_from_migration_step(ctx, migration_step)
-    folder_path = get_odoo_env_path(ctx, odoo_version)
+    env_path = ctx.obj["env_folder_path"]
+    odoo_env_path = get_odoo_env_path(ctx, odoo_version)
 
     command = generate_odoo_command(
         ctx,
         migration_step,
         database=database,
+        update=update,
+        init=init,
         stop_after_init=stop_after_init,
         shell=shell,
-        update_all=update_all,
+        demo=demo,
     )
 
-    image_name = get_docker_image_tag(ctx, odoo_version)
+    return run_container(
-    container_name = get_docker_container_name(ctx, migration_step)
+        get_docker_image_tag(ctx, odoo_version),
-    logger.info(
+        get_docker_container_name(ctx, migration_step),
-        "Launching Odoo Docker container named %s based on image '%s'."
-        % (container_name, image_name)
-    )
-    container = client.containers.run(
-        image_name,
-        name=container_name,
         command=command,
         ports={"8069": 8069, "5432": 5432},
-        volumes=["%s:/container_env/" % (folder_path)],
+        volumes=[
+            "%s:/env/" % (env_path),
+            "%s:/odoo_env/" % (odoo_env_path),
+        ],
         links={"db": "db"},
-        detach=True,
+        detach=detached_container,
         auto_remove=True,
     )
-    logger.info("Container Launched. Command executed : %s" % command)
-    return container
 
 
 def kill_odoo(ctx, migration_step: dict):
-    client = docker.from_env()
+    kill_container(get_docker_container_name(ctx, migration_step))
-    containers = client.containers.list(
-        all=True,
-        filters={"name": get_docker_container_name(ctx, migration_step)},
-    )
-    for container in containers:
-        logger.info(
-            "Stop container %s, based on image '%s'."
-            % (container.name, ",".join(container.image.tags))
-        )
-        container.stop()

odoo_openupgrade_wizard/tools_system.py

@@ -8,8 +8,12 @@ from jinja2 import Template
 from loguru import logger
 from plumbum.cmd import mkdir
 
+from odoo_openupgrade_wizard import templates
 
-def ensure_folder_exists(folder_path: Path, mode: str = False):
+
+def ensure_folder_exists(
+    folder_path: Path, mode: str = False, git_ignore_content: bool = False
+):
     """Create a local folder.
     - directory is created if it doesn't exist.
     - mode is applied if defined.
@@ -22,6 +26,12 @@ def ensure_folder_exists(folder_path: Path, mode: str = False):
         logger.info("Creating folder '%s' ..." % (folder_path))
         mkdir(cmd)
 
+    if git_ignore_content:
+        ensure_file_exists_from_template(
+            folder_path / Path(".gitignore"),
+            templates.GIT_IGNORE_CONTENT,
+        )
+
 
 def ensure_file_exists_from_template(
     file_path: Path, template_name: str, **args

pyproject.toml

@@ -17,7 +17,11 @@ classifiers = [
     "Intended Audience :: Developers",
     "Development Status :: 2 - Pre-Alpha",
     "Operating System :: Unix",
-    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.6",
+    "Programming Language :: Python :: 3.7",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
     "Framework :: Odoo",
 ]
 

tests/__init__.py

@@ -0,0 +1,23 @@
+import logging
+
+from click.testing import CliRunner
+
+from odoo_openupgrade_wizard.cli import main
+
+_logger = logging.getLogger()
+
+
+def assert_result_cli_invoke(result):
+    pass
+
+
+def cli_runner_invoke(cmd):
+    result = CliRunner().invoke(
+        main,
+        cmd,
+        catch_exceptions=False,
+    )
+    if not result.exit_code == 0:
+        _logger.error("exit_code: %s" % result.exit_code)
+        _logger.error("output: %s" % result.output)
+    assert result.exit_code == 0

tests/cli_A_init_test.py

@@ -1,18 +1,17 @@
 import filecmp
 from pathlib import Path
 
-from click.testing import CliRunner
 from plumbum.cmd import mkdir
 
-from odoo_openupgrade_wizard.cli import main
+from . import cli_runner_invoke
 
 
 def test_cli_init():
     output_folder_path = Path("./tests/output_A")
     expected_folder_path = Path("./tests/output_A_expected")
     mkdir([output_folder_path, "--parents"])
-    result = CliRunner().invoke(
+
-        main,
+    cli_runner_invoke(
         [
             "--env-folder=%s" % output_folder_path,
             "init",
@@ -21,10 +20,8 @@ def test_cli_init():
             "--final-release=12.0",
             "--extra-repository="
             "OCA/web,OCA/server-tools,GRAP/grap-odoo-incubator",
-        ],
+        ]
-        catch_exceptions=False,
     )
-    assert result.exit_code == 0
 
     assert filecmp.cmp(
         output_folder_path / Path("config.yml"),

tests/cli_B_01_get_code_test.py

@@ -1,9 +1,8 @@
 from pathlib import Path
 
-from click.testing import CliRunner
 from plumbum.cmd import mkdir
 
-from odoo_openupgrade_wizard.cli import main
+from . import cli_runner_invoke
 
 
 def test_cli_get_code():
@@ -12,8 +11,7 @@ def test_cli_get_code():
 
     # We initialize an env with only one version to avoid to git clone
     # large data
-    CliRunner().invoke(
+    cli_runner_invoke(
-        main,
         [
             "--env-folder=%s" % output_folder_path,
             "init",
@@ -21,19 +19,15 @@ def test_cli_get_code():
             "--initial-release=14.0",
             "--final-release=14.0",
             "--extra-repository=OCA/web",
-        ],
+        ]
-        catch_exceptions=False,
     )
 
-    result = CliRunner().invoke(
+    cli_runner_invoke(
-        main,
         [
             "--env-folder=%s" % output_folder_path,
             "get-code",
-        ],
+        ]
-        catch_exceptions=False,
     )
-    assert result.exit_code == 0
 
     openupgrade_path = output_folder_path / Path(
         "./src/env_14.0/src/openupgrade"

tests/cli_B_02_docker_build_test.py

@@ -1,27 +1,23 @@
 from pathlib import Path
 
 import docker
-from click.testing import CliRunner
 
-from odoo_openupgrade_wizard.cli import main
+from . import cli_runner_invoke
 
 
 def test_cli_docker_build():
     output_folder_path = Path("./tests/output_B")
 
-    result = CliRunner().invoke(
+    cli_runner_invoke(
-        main,
         [
             "--env-folder=%s" % output_folder_path,
             "docker-build",
             "--releases=14.0",
-        ],
+        ]
-        catch_exceptions=False,
     )
-    assert result.exit_code == 0
 
     docker_client = docker.from_env()
 
     assert docker_client.images.get(
-        "odoo-openupgrade-wizard-image-test-cli-14.0"
+        "odoo-openupgrade-wizard-image__test-cli__14.0"
     )

tests/cli_B_03_run_test.py

@@ -1,28 +1,34 @@
 from pathlib import Path
 
-import docker
+from . import cli_runner_invoke
-from click.testing import CliRunner
 
-from odoo_openupgrade_wizard.cli import main
+# import docker
 
 
 def test_cli_run():
     output_folder_path = Path("./tests/output_B")
 
-    result = CliRunner().invoke(
+    db_name = "database_test_cli_run_3"
-        main,
+    cli_runner_invoke(
         [
             "--env-folder=%s" % output_folder_path,
             "run",
             "--step=1",
-            "--duration=2",
+            "--database=%s" % db_name,
-        ],
+            "--init-modules=base",
-        catch_exceptions=False,
+            "--stop-after-init",
+        ]
     )
-    assert result.exit_code == 0
 
-    # Ensure that all the containers are removed
+    # # Ensure that a subfolder filestore/DB_NAME has been created
-    docker_client = docker.from_env()
+    # db_filestore_path = output_folder_path / Path(
-    assert not docker_client.containers.list(
+    #     "./filestore/filestore/%s" % db_name
-        all=True, filters={"name": "odoo-openupgrade-wizard"}
+    # )
-    )
+
+    # assert db_filestore_path.exists()
+
+    # # Ensure that all the containers are removed
+    # docker_client = docker.from_env()
+    # assert not docker_client.containers.list(
+    #     all=True, filters={"name": "odoo-openupgrade-wizard"}
+    # )

tests/output_A_expected/config.yml

@@ -1,5 +1,7 @@
 project_name: test-cli
 
+host_odoo_xmlrpc_port: 8069
+
 odoo_versions:
 
   - release: 9.0
@@ -33,7 +35,7 @@ migration_steps:
     action: upgrade
     complete_name: step_04__upgrade__12.0
 
-  - name: 6
+  - name: 5
     release: 12.0
     action: update
-    complete_name: step_06__update__12.0
+    complete_name: step_05__update__12.0

tox.ini

@@ -1,7 +1,7 @@
 [tox]
 isolated_build = true
 skipsdist = True
-envlist = py36, py37, py38
+envlist = py36, py37, py38, py39, py310
 
 [testenv]
 whitelist_externals = poetry