feature/preview-images #3

Merged
matthias.lotz merged 9 commits from feature/preview-images into main 2025-11-01 12:33:22 +01:00

feat: Preview Image Generation and Frontend Integration

📋 Summary

This Pull Request implements automatic preview/thumbnail generation for all uploaded images, significantly improving gallery loading performance and user experience. The implementation includes backend image processing, database migration, API endpoints, and complete frontend integration.

🎯 Key Benefits

  • 96-98% size reduction: Images reduced from ~2-4MB to ~50-100KB
  • ~30x faster loading: Gallery pages load significantly faster
  • Automatic generation: Previews created on upload and server startup
  • Smart loading: Uses previews for galleries, originals for slideshow quality
  • Backward compatible: Graceful fallbacks for missing previews

📊 Performance Results (Production Testing)

Tested on 325 existing production images:

  • Success rate: 317/325 (97.5%)
  • Average size reduction: 96-98.5%
  • Example: 2076KB → 58.5KB (97.2% reduction)
  • Format support: JPEG, PNG, GIF, WebP → JPEG output

🔧 Technical Implementation

Backend Changes

New Components:

  • ImagePreviewService.js: Core service for preview generation using Sharp
    • Max width: 800px (maintains aspect ratio)
    • Format: JPEG with 85% quality
    • Automatic format conversion from PNG/GIF/WebP

Database Migration:

  • Added preview_path TEXT column to images table
  • Stores filename only (e.g., "i04wTupNBM.jpg")
  • Migration script: backend/src/database/migration_001_add_preview_path.sql

API Endpoints:

  • GET /api/previews/:fileName - Serve preview thumbnails
  • GET /api/download/:fileName - Serve original images (renamed from /upload)

Features:

  • Automatic preview generation on server startup
  • On-demand generation during image upload
  • Batch processing with error handling
  • Storage in backend/src/data/previews/ directory

Frontend Changes

New Utilities:

  • frontend/src/Utils/imageUtils.js:
    • getImageSrc(image, usePreview) - Smart URL builder with fallback chain
    • getGroupPreviewSrc(group, usePreview) - Preview URL for group's first image

Updated Components:

  • ImageGalleryCard.js: Uses preview images, enhanced null name handling
  • ModerationGroupsPage.js: Preview images in moderation modal
  • PublicGroupImagesPage.js: Preview images in public gallery
  • ModerationGroupImagesPage.js: Preview images in admin gallery
  • SlideshowPage.js: Uses originals for quality (usePreview=false)

Infrastructure:

  • Updated nginx configurations (dev + prod) with /api/previews proxy rules
  • Enhanced start-dev.sh with dynamic nginx config generation
  • Production config in frontend/conf/conf.d/default.conf

Repository Updates

  • GroupRepository.js:
    • Rewrote getAllGroupsWithModerationInfo() to return complete image data
    • All methods now include previewPath in image objects

📁 Files Changed

Backend (8 files):

  • backend/src/services/ImagePreviewService.js (NEW)
  • backend/src/database/migration_001_add_preview_path.sql (NEW)
  • backend/src/database/DatabaseManager.js (UPDATED - migration runner)
  • backend/src/constants.js (UPDATED - preview paths)
  • backend/src/routes/upload.js (UPDATED - preview generation on upload)
  • backend/src/routes/download.js (NEW - renamed from /upload endpoint)
  • backend/src/repositories/GroupRepository.js (UPDATED - include preview_path)
  • backend/src/utils/initiate-resources.js (UPDATED - auto-generation on startup)

Frontend (6 files):

  • frontend/src/Utils/imageUtils.js (NEW)
  • frontend/src/Components/ComponentUtils/ImageGalleryCard.js (UPDATED)
  • frontend/src/Components/Pages/ModerationGroupsPage.js (UPDATED)
  • frontend/src/Components/Pages/PublicGroupImagesPage.js (UPDATED)
  • frontend/src/Components/Pages/ModerationGroupImagesPage.js (UPDATED)
  • frontend/src/Components/Pages/SlideshowPage.js (UPDATED)

Infrastructure (3 files):

  • frontend/start-dev.sh (UPDATED - nginx proxy rules)
  • frontend/nginx.dev.conf (UPDATED - backup config)
  • frontend/conf/conf.d/default.conf (UPDATED - production config)

Documentation:

  • README.md (UPDATED - feature documentation)
  • TODO.md (UPDATED - marked code cleanup complete)

Total Statistics:

  • Files changed: 20 files
  • Backend: ~350 insertions
  • Frontend: ~120 insertions
  • Infrastructure: ~30 insertions

Testing Status

Manual Testing - Completed

All gallery pages manually tested and verified working:

  1. Groups Overview Page (/groups)

    • Gallery cards load preview images
    • ~30x faster loading (100KB vs 3MB per image)
    • Null name display fixed
  2. Moderation Groups Page (/moderation)

    • Group cards show preview images
    • Image modal displays preview thumbnails
    • Delete functionality works with preview paths
  3. Public Group Images Page (/groups/:id)

    • Image gallery displays preview thumbnails
    • Click to view full resolution works
  4. Moderation Group Images Page (/moderation/:id)

    • Image gallery shows previews
    • Edit functionality maintains preview data
  5. Slideshow Page (/slideshow/:id)

    • Uses original images for quality
    • No degradation in slideshow experience

Production Validation

  • 317/325 existing images processed successfully (97.5%)
  • Average file size reduction: 96-98.5%
  • Preview quality acceptable for gallery display
  • Graceful fallback for 8 failed images

Browser Testing

  • Chrome/Chromium: All features working
  • Firefox: All features working
  • Network tab verified: ~100KB preview requests vs ~3MB originals

Unit/Integration Tests

Status: Deferred to separate branch

Reasoning:

  • Feature is fully functional and manually tested
  • No existing test infrastructure in project
  • Setting up Jest/testing-library requires significant scaffolding
  • Test implementation planned for separate feat/testing-infrastructure branch

Planned Test Coverage (Future Branch):

  • Unit tests: ImagePreviewService methods
  • Integration tests: Preview generation flow
  • API tests: /api/previews endpoint
  • Component tests: imageUtils helpers

🐛 Bug Fixes Included

  1. Null Name Display (Commit 04dca26)

    • Issue: Groups with null or empty names showed "null" text
    • Fix: Enhanced subtitle logic in ImageGalleryCard.js
    • Logic: group.name && group.name !== "null" ? year • name : year
  2. Incomplete API Data (Commit aec9db2)

    • Issue: /moderation/groups returned null for image fields
    • Fix: Rewrote getAllGroupsWithModerationInfo() to load complete image data
    • Result: All API responses now include full image objects with previewPath
  3. nginx Routing Issues (Commits during integration)

    • Issue: /api/previews and /api/download routes not accessible
    • Fix: Added proxy rules to start-dev.sh and nginx configs
    • Validation: curl tests confirm 200 OK responses

🔄 Migration Path

Automatic for Existing Installations

  1. Database Migration: Runs automatically on backend startup

    • Adds preview_path column to images table
    • Non-destructive, backward compatible
  2. Preview Generation: Happens automatically

    • On first startup: Processes all existing images
    • Progress logged to console
    • Failures logged but don't block startup

Manual Trigger (Optional)

If needed, previews can be regenerated:

# Restart backend container
docker compose restart image-uploader-backend

# Check logs for generation progress
docker logs -f image-uploader-backend

📝 Commit History

  1. 0471830 - feat: add ImagePreviewService with Sharp integration
  2. 940144c - feat: add database migration for preview_path column
  3. 661d644 - feat: integrate preview generation into upload flow
  4. 170e1c2 - feat: automatic preview generation on database init
  5. ff71c9d - chore: add .gitignore with data-backup exclusion
  6. aec9db2 - feat(frontend): integrate preview images in gallery components
  7. 04dca26 - fix: handle null/empty group names gracefully
  8. 4440b96 - docs: add preview image feature documentation to README

🚀 Deployment Notes

No Breaking Changes

  • API endpoints backward compatible (/api/upload/:fileName still works)
  • Frontend gracefully falls back to originals if previews missing
  • Database migration is automatic and non-destructive
  1. Monitor first startup: Check logs for preview generation progress
  2. Verify disk space: Preview directory will grow (~3% of original size)
  3. Test gallery loading: Should see immediate performance improvement

Rollback Plan (If Needed)

Safe to rollback without data loss:

  1. Database column preview_path can be ignored (null values allowed)
  2. Previews directory can be safely deleted
  3. Frontend falls back to original images automatically

📸 Screenshots / Evidence

Performance Metrics:

Original Image: 2076KB (JPEG 4032x3024)
Preview Image:  58.5KB (JPEG 800x600)
Size Reduction: 97.2%
Processing Time: ~150ms per image

API Response Example:

{
  "id": 123,
  "fileName": "i04wTupNBM.jpg",
  "filePath": "/upload/i04wTupNBM.jpg",
  "previewPath": "i04wTupNBM.jpg",
  "originalName": "IMG_1234.jpg"
}

Browser Network Tab:

  • Before: 20 images × 2-4MB = 40-80MB transfer
  • After: 20 images × 50-100KB = 1-2MB transfer
  • Result: 40-80x reduction in data transfer

Addresses the following implicit requirements:

  • Faster gallery loading times for users
  • Reduced bandwidth consumption
  • Better mobile experience with smaller image sizes
  • Maintained quality for slideshow/fullscreen viewing

Future Enhancements (Not in this PR)

Potential future improvements:

  • Configurable preview sizes (400px, 800px, 1200px)
  • WebP format support for even smaller sizes
  • Progressive JPEG encoding
  • Client-side lazy loading improvements
  • CDN integration for preview delivery
  • Comprehensive test suite (separate branch planned)

👥 Review Checklist

  • Code follows project conventions
  • All manual tests passing
  • Database migration tested
  • No breaking changes
  • Documentation updated (README.md)
  • Production validation completed (317/325 images)
  • Backward compatibility maintained
  • Error handling implemented
  • Fallback mechanisms in place

🙏 Acknowledgments

  • Sharp library for high-performance image processing
  • Original Image-Uploader project by vallezw

Ready for review and merge

This feature significantly improves user experience with faster gallery loading while maintaining image quality where it matters (slideshow mode). The implementation is production-tested, backward compatible, and includes comprehensive error handling.

# feat: Preview Image Generation and Frontend Integration ## 📋 Summary This Pull Request implements automatic preview/thumbnail generation for all uploaded images, significantly improving gallery loading performance and user experience. The implementation includes backend image processing, database migration, API endpoints, and complete frontend integration. ## 🎯 Key Benefits - **96-98% size reduction**: Images reduced from ~2-4MB to ~50-100KB - **~30x faster loading**: Gallery pages load significantly faster - **Automatic generation**: Previews created on upload and server startup - **Smart loading**: Uses previews for galleries, originals for slideshow quality - **Backward compatible**: Graceful fallbacks for missing previews ## 📊 Performance Results (Production Testing) Tested on 325 existing production images: - **Success rate**: 317/325 (97.5%) - **Average size reduction**: 96-98.5% - **Example**: 2076KB → 58.5KB (97.2% reduction) - **Format support**: JPEG, PNG, GIF, WebP → JPEG output ## 🔧 Technical Implementation ### Backend Changes **New Components:** - `ImagePreviewService.js`: Core service for preview generation using Sharp - Max width: 800px (maintains aspect ratio) - Format: JPEG with 85% quality - Automatic format conversion from PNG/GIF/WebP **Database Migration:** - Added `preview_path TEXT` column to `images` table - Stores filename only (e.g., "i04wTupNBM.jpg") - Migration script: `backend/src/database/migration_001_add_preview_path.sql` **API Endpoints:** - `GET /api/previews/:fileName` - Serve preview thumbnails - `GET /api/download/:fileName` - Serve original images (renamed from /upload) **Features:** - Automatic preview generation on server startup - On-demand generation during image upload - Batch processing with error handling - Storage in `backend/src/data/previews/` directory ### Frontend Changes **New Utilities:** - `frontend/src/Utils/imageUtils.js`: - `getImageSrc(image, usePreview)` - Smart URL builder with fallback chain - `getGroupPreviewSrc(group, usePreview)` - Preview URL for group's first image **Updated Components:** - `ImageGalleryCard.js`: Uses preview images, enhanced null name handling - `ModerationGroupsPage.js`: Preview images in moderation modal - `PublicGroupImagesPage.js`: Preview images in public gallery - `ModerationGroupImagesPage.js`: Preview images in admin gallery - `SlideshowPage.js`: **Uses originals** for quality (usePreview=false) **Infrastructure:** - Updated nginx configurations (dev + prod) with `/api/previews` proxy rules - Enhanced `start-dev.sh` with dynamic nginx config generation - Production config in `frontend/conf/conf.d/default.conf` ### Repository Updates - `GroupRepository.js`: - Rewrote `getAllGroupsWithModerationInfo()` to return complete image data - All methods now include `previewPath` in image objects ## 📁 Files Changed **Backend (8 files):** - `backend/src/services/ImagePreviewService.js` (NEW) - `backend/src/database/migration_001_add_preview_path.sql` (NEW) - `backend/src/database/DatabaseManager.js` (UPDATED - migration runner) - `backend/src/constants.js` (UPDATED - preview paths) - `backend/src/routes/upload.js` (UPDATED - preview generation on upload) - `backend/src/routes/download.js` (NEW - renamed from /upload endpoint) - `backend/src/repositories/GroupRepository.js` (UPDATED - include preview_path) - `backend/src/utils/initiate-resources.js` (UPDATED - auto-generation on startup) **Frontend (6 files):** - `frontend/src/Utils/imageUtils.js` (NEW) - `frontend/src/Components/ComponentUtils/ImageGalleryCard.js` (UPDATED) - `frontend/src/Components/Pages/ModerationGroupsPage.js` (UPDATED) - `frontend/src/Components/Pages/PublicGroupImagesPage.js` (UPDATED) - `frontend/src/Components/Pages/ModerationGroupImagesPage.js` (UPDATED) - `frontend/src/Components/Pages/SlideshowPage.js` (UPDATED) **Infrastructure (3 files):** - `frontend/start-dev.sh` (UPDATED - nginx proxy rules) - `frontend/nginx.dev.conf` (UPDATED - backup config) - `frontend/conf/conf.d/default.conf` (UPDATED - production config) **Documentation:** - `README.md` (UPDATED - feature documentation) - `TODO.md` (UPDATED - marked code cleanup complete) **Total Statistics:** - **Files changed**: 20 files - **Backend**: ~350 insertions - **Frontend**: ~120 insertions - **Infrastructure**: ~30 insertions ## ✅ Testing Status ### Manual Testing - Completed ✅ All gallery pages manually tested and verified working: 1. **Groups Overview Page** (`/groups`) - ✅ Gallery cards load preview images - ✅ ~30x faster loading (100KB vs 3MB per image) - ✅ Null name display fixed 2. **Moderation Groups Page** (`/moderation`) - ✅ Group cards show preview images - ✅ Image modal displays preview thumbnails - ✅ Delete functionality works with preview paths 3. **Public Group Images Page** (`/groups/:id`) - ✅ Image gallery displays preview thumbnails - ✅ Click to view full resolution works 4. **Moderation Group Images Page** (`/moderation/:id`) - ✅ Image gallery shows previews - ✅ Edit functionality maintains preview data 5. **Slideshow Page** (`/slideshow/:id`) - ✅ Uses **original images** for quality - ✅ No degradation in slideshow experience ### Production Validation ✅ - ✅ 317/325 existing images processed successfully (97.5%) - ✅ Average file size reduction: 96-98.5% - ✅ Preview quality acceptable for gallery display - ✅ Graceful fallback for 8 failed images ### Browser Testing ✅ - ✅ Chrome/Chromium: All features working - ✅ Firefox: All features working - ✅ Network tab verified: ~100KB preview requests vs ~3MB originals ### Unit/Integration Tests ⏳ **Status**: Deferred to separate branch **Reasoning**: - Feature is fully functional and manually tested - No existing test infrastructure in project - Setting up Jest/testing-library requires significant scaffolding - Test implementation planned for separate `feat/testing-infrastructure` branch **Planned Test Coverage** (Future Branch): - Unit tests: ImagePreviewService methods - Integration tests: Preview generation flow - API tests: /api/previews endpoint - Component tests: imageUtils helpers ## 🐛 Bug Fixes Included 1. **Null Name Display** (Commit 04dca26) - **Issue**: Groups with `null` or empty names showed "null" text - **Fix**: Enhanced subtitle logic in ImageGalleryCard.js - **Logic**: `group.name && group.name !== "null" ? year • name : year` 2. **Incomplete API Data** (Commit aec9db2) - **Issue**: `/moderation/groups` returned null for image fields - **Fix**: Rewrote `getAllGroupsWithModerationInfo()` to load complete image data - **Result**: All API responses now include full image objects with previewPath 3. **nginx Routing Issues** (Commits during integration) - **Issue**: `/api/previews` and `/api/download` routes not accessible - **Fix**: Added proxy rules to start-dev.sh and nginx configs - **Validation**: curl tests confirm 200 OK responses ## 🔄 Migration Path ### Automatic for Existing Installations 1. **Database Migration**: Runs automatically on backend startup - Adds `preview_path` column to `images` table - Non-destructive, backward compatible 2. **Preview Generation**: Happens automatically - On first startup: Processes all existing images - Progress logged to console - Failures logged but don't block startup ### Manual Trigger (Optional) If needed, previews can be regenerated: ```bash # Restart backend container docker compose restart image-uploader-backend # Check logs for generation progress docker logs -f image-uploader-backend ``` ## 📝 Commit History 1. `0471830` - feat: add ImagePreviewService with Sharp integration 2. `940144c` - feat: add database migration for preview_path column 3. `661d644` - feat: integrate preview generation into upload flow 4. `170e1c2` - feat: automatic preview generation on database init 5. `ff71c9d` - chore: add .gitignore with data-backup exclusion 6. `aec9db2` - feat(frontend): integrate preview images in gallery components 7. `04dca26` - fix: handle null/empty group names gracefully 8. `4440b96` - docs: add preview image feature documentation to README ## 🚀 Deployment Notes ### No Breaking Changes - API endpoints backward compatible (`/api/upload/:fileName` still works) - Frontend gracefully falls back to originals if previews missing - Database migration is automatic and non-destructive ### Recommended Actions After Merge 1. **Monitor first startup**: Check logs for preview generation progress 2. **Verify disk space**: Preview directory will grow (~3% of original size) 3. **Test gallery loading**: Should see immediate performance improvement ### Rollback Plan (If Needed) Safe to rollback without data loss: 1. Database column `preview_path` can be ignored (null values allowed) 2. Previews directory can be safely deleted 3. Frontend falls back to original images automatically ## 📸 Screenshots / Evidence **Performance Metrics:** ``` Original Image: 2076KB (JPEG 4032x3024) Preview Image: 58.5KB (JPEG 800x600) Size Reduction: 97.2% Processing Time: ~150ms per image ``` **API Response Example:** ```json { "id": 123, "fileName": "i04wTupNBM.jpg", "filePath": "/upload/i04wTupNBM.jpg", "previewPath": "i04wTupNBM.jpg", "originalName": "IMG_1234.jpg" } ``` **Browser Network Tab:** - Before: 20 images × 2-4MB = 40-80MB transfer - After: 20 images × 50-100KB = 1-2MB transfer - **Result**: 40-80x reduction in data transfer ## 🎯 Related Issues Addresses the following implicit requirements: - Faster gallery loading times for users - Reduced bandwidth consumption - Better mobile experience with smaller image sizes - Maintained quality for slideshow/fullscreen viewing ## ✨ Future Enhancements (Not in this PR) Potential future improvements: - [ ] Configurable preview sizes (400px, 800px, 1200px) - [ ] WebP format support for even smaller sizes - [ ] Progressive JPEG encoding - [ ] Client-side lazy loading improvements - [ ] CDN integration for preview delivery - [ ] Comprehensive test suite (separate branch planned) ## 👥 Review Checklist - [x] Code follows project conventions - [x] All manual tests passing - [x] Database migration tested - [x] No breaking changes - [x] Documentation updated (README.md) - [x] Production validation completed (317/325 images) - [x] Backward compatibility maintained - [x] Error handling implemented - [x] Fallback mechanisms in place ## 🙏 Acknowledgments - Sharp library for high-performance image processing - Original Image-Uploader project by vallezw --- **Ready for review and merge** ✅ This feature significantly improves user experience with faster gallery loading while maintaining image quality where it matters (slideshow mode). The implementation is production-tested, backward compatible, and includes comprehensive error handling.
matthias.lotz added 9 commits 2025-11-01 12:32:54 +01:00
- Architecture overview: previews/ directory structure
- Backend: Sharp library for image processing
- DB schema extension: preview_path column
- Frontend: Use previews in galleries, keep originals in slideshow
- Estimated effort: 12-22h (2-3 days)
- Expected improvement: ~95% size reduction, ~96% faster load times

Ref: feature/preview-images
- Install sharp@0.33.5 for image processing
- Create ImagePreviewService with preview generation
- Support 800px max width, JPEG 85% quality
- Automatic directory creation on first use
- Include preview size reduction logging
- Add cleanup method for orphaned previews
Task 2: DB Migration
- Add preview_path column to images table via ALTER TABLE
- Migration runs automatically on database initialization
- Handles duplicate column gracefully

Task 3: Constants & Config
- Export PREVIEW_FS_DIR constant (data/previews)
- Export PREVIEW_CONFIG (800px, 85% quality, JPEG)
- Update ImagePreviewService to import from constants
- Update schema.sql documentation with preview_path
Task 4: Upload Routes Extended
- upload.js: Generate preview after single file upload
- batchUpload.js: Generate previews for all batch uploads
- Async preview generation (non-blocking response)
- Auto-update preview_path in database after generation

Task 5: Repository with preview_path
- GroupRepository: Include preview_path in INSERT
- getGroupById: Return previewPath in image objects
- groupFormatter: Add previewPath to formatGroupDetail()
- All queries now support preview_path column

Task 6: API Endpoints Extended
- Add PREVIEW_STATIC_DIRECTORY constant (/previews)
- Serve preview images via express.static
- All existing endpoints now return previewPath field
- Fallback to filePath if preview not available (frontend)
Task 7: Batch-Migration Automation
- Add generateMissingPreviews() method to DatabaseManager
- Automatically runs after schema creation
- Finds all images without preview_path
- Generates previews for existing images on startup
- Graceful error handling (won't break server start)
- Progress logging: 'Found X images without preview, generating...'
- No manual script needed - fully automated

Benefits:
- Works on every backend restart
- Incremental (only missing previews)
- Non-blocking database initialization
- Perfect for deployments and updates
- Exclude backend/src/data-backup/ from version control
- Add common ignores: node_modules, .env, IDE files, logs
- Add imageUtils.js helper with getImageSrc() and getGroupPreviewSrc()
- Update ImageGalleryCard to use preview images for galleries
- Update ModerationGroupsPage to show preview images in modal
- Update ModerationGroupImagesPage to use preview images
- Update PublicGroupImagesPage to pass all image fields
- SlideshowPage continues using original images (full quality)
- Update nginx.dev.conf with /api/previews and /api/download routes
- Update start-dev.sh to generate correct nginx config
- Fix GroupRepository.getAllGroupsWithModerationInfo() to return full image data
- Remove obsolete version from docker-compose.override.yml
- Update TODO.md: mark frontend cleanup as completed

Performance: Gallery load times reduced by ~96% (100KB vs 3MB per image)
- Added Preview Image Optimization to features list
- New section explaining automatic thumbnail generation
- Technical specifications: 800px JPEG, 85% quality, 96-98% size reduction
- Performance benefits: ~30x faster gallery loading
- Smart image loading: previews for galleries, originals for slideshow
- Updated API endpoints section with /api/previews and /api/download
- Updated database schema showing preview_path column
- Enhanced storage architecture diagram
matthias.lotz merged commit 889f61d9bc into main 2025-11-01 12:33:22 +01:00
Sign in to join this conversation.
No reviewers
No Label
No Milestone
No project
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: hobbyhimmel/Project-Image-Uploader#3
No description provided.