hobbyhimmel/odoo_mqtt
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
a7897ecc9b
odoo_mqtt/IMPLEMENTATION_PLAN.md
matthias.lotz 807436df58 initial version
2026-02-10 20:00:27 +01:00

14 KiB
Raw Blame History

Implementation Plan: IoT Bridge & Odoo Integration

Ziel: Sidecar-Container-Architektur mit periodischer Session-Aggregation
Stand: 03.02.2026
Strategie: Bridge zuerst (standalone testbar), dann Odoo API, dann Integration

📦 Bestandsaufnahme

Vorhanden (wiederverwendbar)

  • ✅ python_prototype/ - Standalone MQTT Client & Session Detector (Basis für Bridge)
  • ✅ services/mqtt_client.py - Odoo-integrierter MQTT Client (Referenz)
  • ✅ services/session_detector.py - State Machine Logik (portierbar)
  • ✅ services/parsers/shelly_parser.py - Shelly PM Parser (direkt übernehmen)
  • ✅ models/ - Odoo Models (anzupassen)
  • ✅ Mosquitto MQTT Broker (läuft)

Neu zu erstellen

  • ✅ iot_bridge/ Container-Source (Skeleton existiert)
  • ❌ Odoo REST API Controller
  • ✅ Session-Aggregation in Bridge
  • ✅ Odoo Session-Model mit Billing-Logik
  • ✅ Docker Compose Integration

🎯 Phase 1: Bridge Container (Standalone)

Ziel: Bridge läuft unabhängig, loggt auf Console, nutzt YAML-Config

1.1 Bridge Grundstruktur ✅

  • iot_bridge/config.yaml erstellen (Device-Registry, MQTT Settings)
  • iot_bridge/config.py - Config Loader (YAML → Dataclass)
  • Mock-OdooClient (gibt hardcoded Config zurück, kein HTTP)
  • Logger Setup (structlog mit JSON output)

Test: ✅ Bridge startet, lädt Config, loggt Status (JSON), Graceful Shutdown funktioniert

1.2 MQTT Client portieren ✅

  • python_prototype/mqtt_client.py → iot_bridge/mqtt_client.py
  • Shelly Parser integrieren (copy from services/parsers/)
  • Connection Handling (reconnect, error handling)
  • Message-Callback registrieren
  • TLS/SSL Support (Port 8883)

Test: ✅ Bridge empfängt Shelly-Messages (40-55W), parst apower, loggt auf Console (JSON), TLS funktioniert Reconnect Test: ✅ Broker neugestartet → Bridge disconnected (rc=7) → Auto-Reconnect → Re-Subscribe → Messages wieder empfangen

1.3 Session Detector mit Aggregation ✅

  • session_detector.py portieren (5-State Machine)
  • Aggregation-Logik hinzufügen:
    • Interner State-Tracker (1s Updates)
    • Timer für heartbeat_interval_s
    • Berechnung: interval_working_s, interval_standby_s
  • Session-IDs generieren (UUID)
  • Events sammeln (event_callback)
  • Unit Tests (9 Tests, alle PASSED)
  • Shelly Simulator für Testing
  • Integration Tests (7 Tests, alle PASSED)
  • Lokaler Mosquitto Container (docker-compose.dev.yaml)

Test: ✅ Session gestartet (34.5W > 20W threshold), State-Machine funktioniert, Heartbeat-Events nach 10s Unit Tests: ✅ 9/9 passed (test_session_detector.py) Integration Tests: ✅ 7/7 passed (test_bridge_integration.py) - Session Start, Heartbeat, Multi-Device, Timeout

1.4 Event Queue & Retry Logic ✅

  • event_queue.py: Event-Queue mit Retry-Logic
    • Exponential Backoff (1s → 2s → 4s → ... → max 60s)
    • Max 10 Retries
    • Background-Thread für Queue-Processing
    • Thread-safe mit collections.deque und threading.Lock
  • event_uid zu allen Events hinzufügen (UUID)
  • Queue-Integration in main.py
    • on_event_generated() nutzt Queue statt direktem send
    • Queue-Start/Stop im Lifecycle
  • Mock-Odoo Failure-Simulation
    • mock_failure_rate in config.yaml (0.0-1.0)
    • MockOdooClient wirft Exceptions bei failures
  • Unit Tests: test_event_queue.py (13 Tests, alle PASSED)
    • Queue Operations (enqueue, statistics)
    • Exponential Backoff Berechnung
    • Retry Logic mit Mock-Callback
    • Max Retries exceeded
  • Integration Tests: test_retry_logic.py (2 Tests, PASSED in 48.29s)
    • test_retry_on_odoo_failure: Events werden enqueued
    • test_eventual_success_after_retries: 50% failure rate → eventual success

Test: ✅ Mock-Odoo-Client gibt 500 → Events in Queue → Retry mit Backoff → Success Unit Tests: ✅ 13/13 passed Integration Tests: ✅ 2/2 passed in 48.29s

1.5 Docker Container Build

  • Multi-stage Dockerfile finalisiert
    • Builder Stage: gcc + pip install dependencies
    • Runtime Stage: minimal image, non-root user (bridge:1000)
    • Python packages korrekt nach /usr/local/lib/python3.11/site-packages kopiert
  • .dockerignore erstellt (venv/, tests/, pycache, logs/, data/)
  • requirements.txt erweitert mit PyYAML>=6.0
  • docker build -t iot_mqtt_bridge:latest
    • 3 Build-Iterationen (Package-Path-Fix erforderlich)
    • Finale Image: 8b690d20b5f7
  • docker run erfolgreich getestet
    • Volume mount: config.yaml (read-only)
    • Network: host (für MQTT-Zugriff)
    • Container verbindet sich mit mqtt.majufilo.eu:8883
    • Sessions werden erkannt und Events verarbeitet
  • Development Setup
    • config.yaml.dev für lokale Tests (Mosquitto + Odoo)
    • docker-compose.dev.yaml erweitert mit iot-bridge service

Test: ✅ Container läuft produktiv, empfängt MQTT, erkennt Sessions Docker: ✅ Multi-stage build, 71MB final image, non-root user

🎯 Phase 2: Odoo REST API

Ziel: Odoo REST API für Bridge-Config und Event-Empfang

2.1 Models anpassen

  • ows.iot.event Model erstellt (models/iot_event.py)
    • event_uid (unique constraint)
    • event_type (selection: session_started/updated/stopped/timeout/heartbeat/power_change)
    • payload_json (JSON Field)
    • device_id, session_id (Char fields)
    • Auto-Linking zu mqtt.device und mqtt.session
    • Payload-Extraktion: power_w, state
    • processed flag, processing_error für Error-Tracking
  • mqtt.device erweitert:
    • device_id (External ID für Bridge API)
    • Bestehende Felder: strategy_config (JSON), session_strategy, parser_type, topic_pattern
  • mqtt.session - bereits vorhanden:
    • session_id (external UUID)
    • Duration fields: total_duration_s, standby_duration_s, working_duration_s
    • State tracking: status (running/completed), current_state
    • Power tracking: start_power_w, end_power_w, current_power_w

Test: ✅ Models erstellt, Security Rules aktualisiert

2.2 REST API Controller

  • controllers/iot_api.py erstellt
  • GET /ows/iot/config:
    • Returns: Alle aktiven Devices mit session_config als JSON
    • Auth: public (später API-Key möglich)
    • Response Format: 
      {
  "status": "success",
  "devices": [{
    "device_id": "...",
    "mqtt_topic": "...",
    "parser_type": "...",
    "machine_name": "...",
    "session_config": { strategy, thresholds, ... }
  }],
  "timestamp": "ISO8601"
}
  • POST /ows/iot/event:
    • Schema-Validation (event_type, device_id, event_uid, timestamp required)
    • Event-UID Duplikat-Check → 409 Conflict (idempotent)
    • Event speichern in ows.iot.event
    • Auto-Processing: Session erstellen/updaten/beenden
    • Response Codes: 201 Created, 409 Duplicate, 400 Bad Request, 500 Error
    • JSON-RPC Format (Odoo type='json')

Test: ✅ Controller erstellt, bereit für manuellen Test

curl http://localhost:8069/ows/iot/config
curl -X POST http://localhost:8069/ows/iot/event -d '{...}'

2.3 Bridge Client - OdooClient

  • OdooClient implementiert (iot_bridge/odoo_client.py)
    • get_config(): GET /ows/iot/config (HTTP)
    • send_event(): POST /ows/iot/event (JSON-RPC)
    • Retry-Logic über EventQueue (bereits in Phase 1.4)
    • Duplicate Handling: 409 wird als Success behandelt
    • Error Handling: Exceptions für 4xx/5xx
    • HTTP Session mit requests library
  • config.py erweitert:
    • OdooConfig: base_url, database, username, api_key
    • Entfernt: alte url, token Felder
  • main.py angepasst:
    • OdooClient Initialisierung mit neuen Parametern
    • Conditional: MockOdooClient (use_mock=true) oder OdooClient (use_mock=false)

Test: ✅ Code fertig, bereit für Integration Test

2.4 Integration Testing

  • Odoo Modul upgraden (Models + Controller laden)
    • ✅ Module installed in OWS_MQTT database
    • ✅ Deprecation warnings (non-blocking, documented)
  • mqtt.device angelegt: "Shaper Origin" (device_id: shellypmminig3-48f6eeb73a1c)
    • ✅ Widget fix: CodeEditor 'ace' → 'text' (Odoo 18 compatibility)
    • ✅ connection_id made optional (deprecated legacy field)
  • API manuell getestet
    • ✅ GET /ows/iot/config → HTTP 200, returns device list
    • ✅ POST /ows/iot/event → HTTP 200, creates event records
  • config.yaml.dev: use_mock=false gesetzt
  • Docker Compose gestartet: Odoo + Mosquitto + Bridge
  • End-to-End Tests:
    • ✅ MQTT → Bridge → Odoo event flow working
    • ✅ Bridge fetches config from Odoo (1 device)
    • ✅ Events stored in ows_iot_event table
    • ✅ session_started creates mqtt.session automatically
    • ✅ session_heartbeat updates session (controller fix applied)
  • Odoo 18 Compatibility Fixes:
    • ✅ type='json' routes: Use function parameters instead of request.jsonrequest
    • ✅ event_type Selection: Added 'session_heartbeat'
    • ✅ Timestamp parsing: ISO format → Odoo datetime format
    • ✅ Multiple databases issue: Deleted extra DBs (hh18, odoo18) for db_monodb()
    • ✅ auth='none' working with single database

Outstanding Issues:

  • UI Cleanup - MQTT Connection (deprecated) view removal
    • ✅ Menu items hidden (commented out in mqtt_menus.xml)
    • ✅ mqtt.message menu hidden (replaced by ows.iot.event)
    • ✅ New "IoT Events" menu added
  • UI Cleanup - mqtt.device form:
    • ✅ connection_id field hidden (invisible="1")
    • ✅ device_id field prominent ("External Device ID")
    • ✅ Parser Type + Session Strategy fields clarified
    • ✅ Help text: "Configuration is sent to IoT Bridge via REST API"
    • ✅ Strategy config placeholder updated with all required fields
  • ows.iot.event Views created:
    • ✅ List view with filters (event type, processed status)
    • ✅ Form view with payload display
    • ✅ Search/filters: event type, device, session, date grouping
    • ✅ Security rules configured
  • Create Test Device in Odoo:
    • ✅ "Test Device - Manual Simulation" (device_id: test-device-manual)
    • ✅ Low thresholds for easy testing (10W standby, 50W working)
    • ✅ Test script created: tests/send_test_event.sh
  • Verify session auto-create/update workflow:
    • ✅ session_started creates mqtt.session
    • ✅ session_heartbeat updates durations + power
    • ✅ total_duration_s = total_working_s + total_standby_s (calculated in controller)

Next Steps:

  • UI Testing: Login to Odoo and verify:
    • MQTT -> Devices: See "Shaper Origin" + "Test Device"
    • MQTT -> Sessions: Check duration calculations (Total = Working + Standby)
    • MQTT -> IoT Events: Verify event list, filters work
    • No "Connections" or "Messages" menus visible
  • Duration Field Validation:
    • Check if Total Hours displays correctly (computed from _s fields)
    • Verify Standby Hours + Working Hours sum equals Total Hours

Test: ⏳ Core functionality working, UI cleanup needed

🎯 Phase 3: Integration & End-to-End

Ziel: Bridge ↔ Odoo kommunizieren, Docker Compose Setup

3.1 Bridge: Odoo Client implementieren

  • iot_bridge/odoo_client.py:
    • get_config() → HTTP GET zu Odoo
    • send_event() → HTTP POST zu Odoo
    • Error Handling (401, 409, 500)
  • Config-Refresh alle 5 Min (von Odoo laden)
  • ENV-Variablen: ODOO_URL, MQTT_URL

Test: Bridge holt Config von Odoo, sendet Events → Odoo empfängt

3.2 Docker Compose Setup

  • docker-compose.yaml updaten:
    • Service: iot_bridge
    • Depends on: odoo, mosquitto
    • ENV: ODOO_URL=http://odoo:8069
  • .env.example mit Variablen
  • README Update: Setup-Anleitung

Test: docker compose up -d → Bridge startet → Config von Odoo → Events in DB

3.3 End-to-End Tests

  • Shelly PM einschalten → Session erscheint in Odoo
  • Mehrere State-Wechsel → Heartbeat aggregiert korrekt
  • Bridge Restart → Sessions werden recovered
  • Odoo Config ändern → Bridge lädt neu

Test: Real-World-Szenario mit echter Hardware durchspielen

🎯 Phase 4: Polishing & Dokumentation

4.1 Error Handling & Monitoring

  • Bridge: Structured Logging (JSON)
  • Odoo: Event-Processing Errors loggen
  • Metriken: Events sent/failed, Session count
  • Health-Checks für beide Services

4.2 Dokumentation

  • iot_bridge/README.md aktualisieren (ENV vars, Config)
  • DEPLOYMENT.md - Produktiv-Setup Guide
  • API Docs - REST Endpoints dokumentieren
  • Troubleshooting Guide

📊 Dependency Graph

Phase 1.1-1.4 (Bridge Core)
     ↓
Phase 1.5 (Docker)
     ↓
Phase 2 (Odoo API) ← kann parallel zu 1.1-1.4
     ↓
Phase 3.1 (Integration)
     ↓
Phase 3.2-3.3 (E2E)
     ↓
Phase 4 (Polish)

🚀 Quick Start (nächster Schritt)

Jetzt starten mit:

  1. Phase 1.1: iot_bridge/config.yaml erstellen
  2. Phase 1.2: MQTT Client portieren
  3. Test: Bridge empfängt Shelly-Messages

Empfohlene Reihenfolge:

  • Phase 1 komplett durchziehen (Bridge standalone funktionsfähig)
  • Phase 2 parallel starten (Odoo API)
  • Phase 3 Integration (wenn beides fertig)

✅ Definition of Done

Phase 1 Done:

  • Bridge läuft als Docker Container
  • Empfängt Shelly-Messages
  • State-Detection + Aggregation funktioniert
  • Loggt aggregierte Events auf Console

Phase 2 Done:

  • Odoo REST API antwortet
  • Events werden in DB gespeichert
  • Sessions werden erstellt/aktualisiert
  • Billing Units werden berechnet

Phase 3 Done:

  • Bridge sendet Events an Odoo
  • Docker Compose startet alles zusammen
  • End-to-End Test erfolgreich

Phase 4 Done:

  • Dokumentation vollständig
  • Error Handling robust
  • Produktiv-Ready