geutebruck-api/specs/001-surveillance-api/plan.md

# Implementation Plan: Geutebruck Surveillance API

**Branch**: `001-surveillance-api` | **Date**: 2025-12-08 | **Spec**: [spec.md](./spec.md)
**Input**: Feature specification from `/specs/001-surveillance-api/spec.md`

## Summary

Build a production-ready REST API for Geutebruck GeViScope/GeViSoft video surveillance systems, enabling developers to integrate surveillance capabilities into custom applications without direct SDK complexity. The system uses a C# gRPC bridge to interface with the GeViScope SDK, exposing clean REST/WebSocket endpoints through Python FastAPI.

**Technical Approach**: Python FastAPI + C# gRPC SDK Bridge + GeViScope SDK → delivers <200ms API responses, supports 100+ concurrent video streams, and handles 1000+ WebSocket event subscribers.

## Technical Context

**Language/Version**: Python 3.11+, C# .NET Framework 4.8 (SDK bridge), C# .NET 8.0 (gRPC service)
**Primary Dependencies**:
- **Python**: FastAPI, Uvicorn, SQLAlchemy, Redis (aioredis), protobuf, grpcio, PyJWT, asyncio
- **C#**: GeViScope SDK (GeViProcAPINET_4_0.dll), Grpc.Core, Google.Protobuf
**Storage**: PostgreSQL 14+ (user management, session storage, audit logs), Redis 6.0+ (session cache, pub/sub for WebSocket events)
**Testing**: pytest (Python), xUnit (.NET), 80% minimum coverage, TDD enforced
**Target Platform**: Windows Server 2016+ (SDK bridge + GeViServer), Linux (FastAPI server - optional)
**Project Type**: Web (backend API + SDK bridge service)
**Performance Goals**:
- <200ms p95 for metadata queries (camera lists, status)
- <2s stream initialization
- <100ms event notification delivery
- 100+ concurrent video streams
- 1000+ concurrent WebSocket connections
**Constraints**:
- SDK requires Windows x86 (32-bit) runtime
- Visual C++ 2010 Redistributable (x86) mandatory
- Full GeViSoft installation required (not just SDK)
- GeViServer must be running on network-accessible host
- All SDK operations must use Channel-based architecture
**Scale/Scope**:
- Support 50+ cameras per installation
- Handle 10k+ events/hour during peak activity
- Store 90 days audit logs (configurable)
- Support 100+ concurrent operators

## Constitution Check

*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*

### Constitution Alignment

✅ **Single Source of Truth**: OpenAPI spec serves as the contract, auto-generated from code
✅ **Test-First Development**: TDD enforced with pytest/xUnit, 80% minimum coverage
✅ **Simplicity**: REST over custom protocols, JWT over session cookies, direct stream URLs over proxying
✅ **Clear Abstractions**: SDK Bridge isolates SDK complexity from Python API layer
✅ **Error Handling**: SDK errors translated to HTTP status codes with user-friendly messages
✅ **Documentation**: Auto-generated OpenAPI docs at `/docs`, quickstart guide provided
✅ **Security First**: JWT authentication, RBAC, rate limiting, audit logging, TLS enforcement

### Exceptions to Constitution

None. All design decisions align with constitution principles.

## Project Structure

### Documentation (this feature)

```text
specs/001-surveillance-api/
├── plan.md              # This file (implementation plan)
├── spec.md              # Feature specification (user stories, requirements)
├── research.md          # Phase 0 output (technical research, architectural decisions)
├── data-model.md        # Phase 1 output (entity schemas, relationships, validation)
├── quickstart.md        # Phase 1 output (developer quick start guide)
├── contracts/           # Phase 1 output (API contracts)
│   └── openapi.yaml     # Complete OpenAPI 3.0 specification
└── tasks.md             # Phase 2 output (will be generated by /speckit.tasks)
```

### Source Code (repository root)

```text
geutebruck-api/
├── src/
│   ├── api/                      # Python FastAPI application
│   │   ├── main.py               # FastAPI app entry point
│   │   ├── config.py             # Configuration management (env vars)
│   │   ├── models/               # SQLAlchemy ORM models
│   │   │   ├── user.py
│   │   │   ├── camera.py
│   │   │   ├── event.py
│   │   │   └── audit_log.py
│   │   ├── schemas/              # Pydantic request/response models
│   │   │   ├── auth.py
│   │   │   ├── camera.py
│   │   │   ├── stream.py
│   │   │   ├── event.py
│   │   │   └── recording.py
│   │   ├── routers/              # FastAPI route handlers
│   │   │   ├── auth.py           # /api/v1/auth/*
│   │   │   ├── cameras.py        # /api/v1/cameras/*
│   │   │   ├── events.py         # /api/v1/events/*
│   │   │   ├── recordings.py     # /api/v1/recordings/*
│   │   │   ├── analytics.py      # /api/v1/analytics/*
│   │   │   └── system.py         # /api/v1/health, /status
│   │   ├── services/             # Business logic layer
│   │   │   ├── auth_service.py
│   │   │   ├── camera_service.py
│   │   │   ├── stream_service.py
│   │   │   ├── event_service.py
│   │   │   └── recording_service.py
│   │   ├── clients/              # External service clients
│   │   │   ├── sdk_bridge_client.py  # gRPC client for SDK bridge
│   │   │   └── redis_client.py       # Redis connection pooling
│   │   ├── middleware/           # FastAPI middleware
│   │   │   ├── auth_middleware.py
│   │   │   ├── rate_limiter.py
│   │   │   └── error_handler.py
│   │   ├── websocket/            # WebSocket event streaming
│   │   │   ├── connection_manager.py
│   │   │   └── event_broadcaster.py
│   │   ├── utils/                # Utility functions
│   │   │   ├── jwt_utils.py
│   │   │   └── error_translation.py
│   │   └── migrations/           # Alembic database migrations
│   │       └── versions/
│   │
│   └── sdk-bridge/               # C# gRPC service (SDK wrapper)
│       ├── GeViScopeBridge.sln
│       ├── GeViScopeBridge/
│       │   ├── Program.cs        # gRPC server entry point
│       │   ├── Services/
│       │   │   ├── CameraService.cs      # Camera operations
│       │   │   ├── StreamService.cs      # Stream management
│       │   │   ├── EventService.cs       # Event subscriptions
│       │   │   ├── RecordingService.cs   # Recording management
│       │   │   └── AnalyticsService.cs   # Analytics configuration
│       │   ├── SDK/
│       │   │   ├── GeViDatabaseWrapper.cs
│       │   │   ├── StateQueryHandler.cs
│       │   │   ├── DatabaseQueryHandler.cs
│       │   │   └── ActionDispatcher.cs
│       │   ├── Models/           # Internal data models
│       │   └── Utils/
│       └── Protos/               # gRPC protocol definitions
│           ├── camera.proto
│           ├── stream.proto
│           ├── event.proto
│           ├── recording.proto
│           └── analytics.proto
│
├── tests/
│   ├── api/
│   │   ├── unit/                 # Unit tests for Python services
│   │   │   ├── test_auth_service.py
│   │   │   ├── test_camera_service.py
│   │   │   └── test_event_service.py
│   │   ├── integration/          # Integration tests with SDK bridge
│   │   │   ├── test_camera_operations.py
│   │   │   ├── test_stream_lifecycle.py
│   │   │   └── test_event_notifications.py
│   │   └── contract/             # OpenAPI contract validation
│   │       └── test_openapi_compliance.py
│   │
│   └── sdk-bridge/
│       ├── Unit/                 # C# unit tests
│       │   ├── CameraServiceTests.cs
│       │   └── StateQueryTests.cs
│       └── Integration/          # Tests with actual SDK
│           └── SdkIntegrationTests.cs
│
├── docs/
│   ├── architecture.md           # System architecture diagram
│   ├── sdk-integration.md        # SDK integration patterns
│   └── deployment.md             # Production deployment guide
│
├── scripts/
│   ├── setup_dev_environment.ps1 # Development environment setup
│   ├── start_services.ps1        # Start all services (Redis, SDK Bridge, API)
│   └── run_tests.sh              # Test execution script
│
├── .env.example                  # Environment variable template
├── requirements.txt              # Python dependencies
├── pyproject.toml                # Python project configuration
├── alembic.ini                   # Database migration configuration
└── README.md                     # Project overview
```

**Structure Decision**: Web application structure selected (backend API + SDK bridge service) because:
1. SDK requires Windows runtime → isolated C# bridge service
2. API layer can run on Linux → flexibility for deployment
3. Clear separation between SDK complexity and API logic
4. gRPC provides high-performance, typed communication between layers
5. Python layer handles web concerns (HTTP, WebSocket, auth, validation)

## Phase 0 - Research ✅ COMPLETED

**Deliverable**: [research.md](./research.md)

**Key Decisions**:
1. **SDK Integration Method**: C# gRPC bridge service (not pythonnet, subprocess, or COM)
   - Rationale: Isolates SDK crashes, maintains type safety, enables independent scaling
2. **Stream Architecture**: Direct RTSP URLs with token authentication (not API proxy)
   - Rationale: Reduces API latency, leverages existing streaming infrastructure
3. **Event Distribution**: FastAPI WebSocket + Redis Pub/Sub
   - Rationale: Supports 1000+ concurrent connections, horizontal scaling capability
4. **Authentication**: JWT with Redis session storage
   - Rationale: Stateless validation, flexible permissions, Redis for quick invalidation
5. **Performance Strategy**: Async Python + gRPC connection pooling
   - Rationale: Non-blocking I/O for concurrent operations, <200ms response targets

**Critical Discoveries**:
- Visual C++ 2010 Redistributable (x86) mandatory for SDK DLL loading
- Full GeViSoft installation required (not just SDK)
- Windows Forms context needed for mixed-mode C++/CLI assemblies
- GeViServer ports: 7700, 7701, 7703 (NOT 7707 as initially assumed)
- SDK connection pattern: Create → RegisterCallback → Connect (order matters!)
- State Queries use GetFirst/GetNext iteration for enumerating entities

See [SDK_INTEGRATION_LESSONS.md](../../SDK_INTEGRATION_LESSONS.md) for complete details.

## Phase 1 - Design ✅ COMPLETED

**Deliverables**:
- [data-model.md](./data-model.md) - Entity schemas, relationships, validation rules
- [contracts/openapi.yaml](./contracts/openapi.yaml) - Complete REST API specification
- [quickstart.md](./quickstart.md) - Developer quick start guide

**Key Components**:

### Data Model
- **User**: Authentication, RBAC (viewer/operator/administrator), permissions
- **Camera**: Channel-based, capabilities (PTZ, analytics), status tracking
- **Stream**: Active sessions with token-authenticated URLs
- **Event**: Surveillance occurrences (motion, alarms, analytics)
- **Recording**: Video segments with ring buffer management
- **AnalyticsConfig**: VMD, NPR, OBTRACK configuration per camera

### API Endpoints (RESTful)
- `POST /api/v1/auth/login` - Authenticate and get JWT tokens
- `POST /api/v1/auth/refresh` - Refresh access token
- `POST /api/v1/auth/logout` - Invalidate tokens
- `GET /api/v1/cameras` - List cameras with filtering
- `GET /api/v1/cameras/{id}` - Get camera details
- `POST /api/v1/cameras/{id}/stream` - Start video stream
- `DELETE /api/v1/cameras/{id}/stream/{stream_id}` - Stop stream
- `POST /api/v1/cameras/{id}/ptz` - PTZ control commands
- `WS /api/v1/events/stream` - WebSocket event notifications
- `GET /api/v1/events` - Query event history
- `GET /api/v1/recordings` - Query recordings
- `POST /api/v1/recordings/{id}/export` - Export video segment
- `GET /api/v1/analytics/{camera_id}` - Get analytics configuration
- `POST /api/v1/analytics/{camera_id}` - Configure analytics
- `GET /api/v1/health` - System health check
- `GET /api/v1/status` - Detailed system status

### gRPC Service Definitions
- **CameraService**: ListCameras, GetCameraDetails, GetCameraStatus
- **StreamService**: StartStream, StopStream, GetStreamStatus
- **PTZService**: MoveCamera, SetPreset, GotoPreset
- **EventService**: SubscribeEvents, UnsubscribeEvents (server streaming)
- **RecordingService**: QueryRecordings, StartRecording, StopRecording
- **AnalyticsService**: ConfigureAnalytics, GetAnalyticsConfig

## Phase 2 - Configuration Management ✅ COMPLETED (2025-12-16)

**Implemented**: GeViSoft configuration management via REST API and gRPC SDK Bridge

**Deliverables**:
- G-Core Server CRUD operations (CREATE, READ, DELETE working; UPDATE has known bug)
- Action Mapping CRUD operations (CREATE, READ, UPDATE, DELETE all working)
- SetupClient integration for configuration download/upload
- Configuration tree parsing and navigation
- Critical bug fixes (cascade deletion prevention)

**Key Components Implemented**:

### REST API Endpoints
- `GET /api/v1/configuration/servers` - List all G-Core servers
- `GET /api/v1/configuration/servers/{server_id}` - Get single server
- `POST /api/v1/configuration/servers` - Create new server
- `PUT /api/v1/configuration/servers/{server_id}` - Update server (⚠️ known bug)
- `DELETE /api/v1/configuration/servers/{server_id}` - Delete server
- `GET /api/v1/configuration/action-mappings` - List all action mappings
- `GET /api/v1/configuration/action-mappings/{mapping_id}` - Get single mapping
- `POST /api/v1/configuration/action-mappings` - Create mapping
- `PUT /api/v1/configuration/action-mappings/{mapping_id}` - Update mapping
- `DELETE /api/v1/configuration/action-mappings/{mapping_id}` - Delete mapping

### gRPC SDK Bridge Implementation
- **ConfigurationService**: Complete CRUD operations for servers and action mappings
- **SetupClient Integration**: Download/upload .set configuration files
- **FolderTreeParser**: Parse GeViSoft binary configuration format
- **FolderTreeWriter**: Write configuration changes back to GeViSoft

### Critical Fixes
- **Cascade Deletion Bug** (2025-12-16): Fixed critical bug where deleting multiple action mappings in ascending order caused ID shifting, resulting in deletion of wrong mappings
  - **Solution**: Always delete in reverse order (highest ID first)
  - **Impact**: Prevented data loss of ~54 mappings during testing
  - **Documentation**: CRITICAL_BUG_FIX_DELETE.md

### Test Scripts
- `comprehensive_crud_test.py` - Full CRUD verification with server and mapping operations
- `safe_delete_test.py` - Minimal test to verify cascade deletion fix
- `server_manager.py` - Production-ready server lifecycle management
- `cleanup_to_base.py` - Restore configuration to base state
- `verify_config_via_grpc.py` - Configuration verification tool

### Known Issues
- Server UPDATE operation fails with "Server ID is required" error (documented, workaround: delete and recreate)
- Bool fields stored as int32 in GeViSoft configuration (acceptable - GeViSet reads correctly)

**Documentation**:
- [SERVER_CRUD_IMPLEMENTATION.md](../../SERVER_CRUD_IMPLEMENTATION.md) - Complete implementation guide
- [CRITICAL_BUG_FIX_DELETE.md](../../CRITICAL_BUG_FIX_DELETE.md) - Cascade deletion bug analysis

**Next**: Phase 3 - Implement remaining user stories (streams, events, analytics)

## Phase 3 - Tasks ⏭️ NEXT

**Command**: `/speckit.tasks`

Will generate:
- Task breakdown with dependencies
- Implementation order (TDD-first)
- Test plan for each task
- Acceptance criteria per task
- Time estimates

**Expected Task Categories**:
1. **Infrastructure Setup**: Repository structure, development environment, CI/CD
2. **SDK Bridge Foundation**: gRPC server, SDK wrapper, basic camera queries
3. **API Foundation**: FastAPI app, authentication, middleware
4. **Core Features**: Camera management, stream lifecycle, event notifications
5. **Extended Features**: Recording management, analytics configuration
6. **Testing & Documentation**: Contract tests, integration tests, deployment docs

## Phase 4 - Implementation ⏭️ FUTURE

**Command**: `/speckit.implement`

Will execute TDD implementation:
- Red: Write failing test
- Green: Minimal code to pass test
- Refactor: Clean up while maintaining passing tests
- Repeat for each task

## Complexity Tracking

No constitution violations. All design decisions follow simplicity and clarity principles:
- ✅ REST over custom protocols
- ✅ JWT over session management
- ✅ Direct streaming over proxying
- ✅ Clear layer separation (API ↔ Bridge ↔ SDK)
- ✅ Standard patterns (FastAPI, gRPC, SQLAlchemy)

## Technology Stack Summary

### Python API Layer
- **Web Framework**: FastAPI 0.104+
- **ASGI Server**: Uvicorn with uvloop
- **ORM**: SQLAlchemy 2.0+
- **Database**: PostgreSQL 14+
- **Cache/PubSub**: Redis 6.0+ (aioredis)
- **Authentication**: PyJWT, passlib (bcrypt)
- **gRPC Client**: grpcio, protobuf
- **Validation**: Pydantic v2
- **Testing**: pytest, pytest-asyncio, httpx
- **Code Quality**: ruff (linting), black (formatting), mypy (type checking)

### C# SDK Bridge
- **Framework**: .NET Framework 4.8 (SDK runtime), .NET 8.0 (gRPC service)
- **gRPC**: Grpc.Core, Grpc.Tools
- **SDK**: GeViScope SDK 7.9.975.68+ (GeViProcAPINET_4_0.dll)
- **Testing**: xUnit, Moq
- **Logging**: Serilog

### Infrastructure
- **Database**: PostgreSQL 14+ (user data, audit logs)
- **Cache**: Redis 6.0+ (sessions, pub/sub)
- **Deployment**: Docker (API layer), Windows Service (SDK bridge)
- **CI/CD**: GitHub Actions
- **Monitoring**: Prometheus metrics, Grafana dashboards

## Commands Reference

### Development
```bash
# Setup environment
.\scripts\setup_dev_environment.ps1

# Start all services
.\scripts\start_services.ps1

# Run API server (development)
cd src/api
uvicorn main:app --reload --host 0.0.0.0 --port 8000

# Run SDK bridge (development)
cd src/sdk-bridge
dotnet run --configuration Debug

# Run tests
pytest tests/api -v --cov=src/api --cov-report=html  # Python
dotnet test tests/sdk-bridge/                         # C#

# Format code
ruff check src/api --fix    # Python linting
black src/api               # Python formatting

# Database migrations
alembic upgrade head        # Apply migrations
alembic revision --autogenerate -m "description"  # Create migration
```

### API Usage
```bash
# Authenticate
curl -X POST http://localhost:8000/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username": "sysadmin", "password": "masterkey"}'

# List cameras
curl -X GET http://localhost:8000/api/v1/cameras \
  -H "Authorization: Bearer YOUR_TOKEN"

# Start stream
curl -X POST http://localhost:8000/api/v1/cameras/{id}/stream \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"resolution": {"width": 1920, "height": 1080, "fps": 30}, "format": "h264"}'

# WebSocket events (Python)
import websockets
uri = f"ws://localhost:8000/api/v1/events/stream?token={TOKEN}"
async with websockets.connect(uri) as ws:
    await ws.send('{"action": "subscribe", "filters": {"event_types": ["motion_detected"]}}')
    while True:
        event = await ws.recv()
        print(event)
```

## Next Steps

1. **Run `/speckit.tasks`** to generate Phase 2 task breakdown
2. **Review tasks** for sequencing and dependencies
3. **Execute `/speckit.implement`** to begin TDD implementation
4. **Iterate** through tasks following Red-Green-Refactor cycle

## References

### Project Documentation
- **Specification**: [spec.md](./spec.md) - User stories, requirements, success criteria
- **Research**: [research.md](./research.md) - Technical decisions and architectural analysis
- **Data Model**: [data-model.md](./data-model.md) - Entity schemas and relationships
- **API Contract**: [contracts/openapi.yaml](./contracts/openapi.yaml) - Complete REST API spec
- **Quick Start**: [quickstart.md](./quickstart.md) - Developer onboarding guide
- **SDK Lessons**: [../../SDK_INTEGRATION_LESSONS.md](../../SDK_INTEGRATION_LESSONS.md) - Critical SDK integration knowledge
- **Constitution**: [../../.specify/memory/constitution.md](../../.specify/memory/constitution.md) - Development principles

### SDK Documentation (Extracted & Searchable)
**Location**: `C:\Gevisoft\Documentation\extracted_html\`

- **Comprehensive SDK Reference**: `C:\DEV\COPILOT\gevisoft-sdk-reference.md`
  - Complete guide to GeViSoft .NET SDK
  - Action mapping implementation patterns
  - Code examples and best practices
  - Generated: 2025-12-11

**Key Documentation Files**:
- **Action Mapping**: `GeViSoft_SDK_Documentation\313Action Mapping.htm`
- **State Queries**: `GeViSoft_SDK_Documentation\414StateQueries.htm`
- **Database Queries**: `GeViSoft_SDK_Documentation\415DatabaseQueries.htm`
- **GeViAPIClient Reference**: `GeViSoft_API_Documentation\class_ge_vi_a_p_i_client.html`
- **CAction Reference**: `GeViSoft_API_Documentation\class_ge_vi_a_p_i___namespace_1_1_c_action.html`

---

**Plan Status**: Phase 0 ✅ | Phase 1 ✅ | Phase 2 ⏭️ | Phase 3 🔄 IN PROGRESS (Configuration Management ✅)
**Last Updated**: 2025-12-16