# GeViSet File Format Reverse Engineering Specification

**Version:** 1.0
**Date:** 2024-12-12
**Status:** In Progress

## Overview

This specification documents the reverse engineering effort to fully parse, understand, and manipulate the GeViSoft `.set` configuration file format. The goal is to enable programmatic reading, editing, and writing of GeViServer configurations, particularly action mappings.

## Background

### What is a .set File?

- **Source**: Exported from GeViSet application or read via `GeViAPI_SetupClient_ReadSetup`
- **Purpose**: Complete GeViServer configuration (cameras, alarms, action mappings, users, etc.)
- **Format**: Proprietary binary format with "GeViSoft Parameters" header
- **Size**: Typically 200-300 KB for production configurations
- **Use Case**: Backup, migration, and programmatic configuration management

### Current State

**What Works:**
- ✅ Read .set file from GeViServer via SetupClient API
- ✅ Extract 64 action mappings from binary data
- ✅ Write back byte-for-byte identical (round-trip verified)
- ✅ Password encryption with `GeViAPI_EncodeString`

**What's Missing:**
- ❌ Full structure parsing (all sections, all items)
- ❌ Understanding of all data types and relationships
- ❌ Ability to add NEW action mappings
- ❌ JSON representation of complete structure
- ❌ Comprehensive Excel export/import

## Requirements

### Primary Request

> "I want to do full reverse engineering - I need to parse the whole file and maybe to json format in the first phase and then we will revert this json or its parts to excel"

### Key Requirements

1. **Parse Entire File Structure**
   - All sections (Alarms, Clients, GeViIO, Cameras, ActionMappings, etc.)
   - All configuration items (key-value pairs)
   - All rules and triggers
   - All metadata and relationships

2. **JSON Serialization**
   - Complete structure in JSON format
   - Human-readable and editable
   - Preserves all data and relationships
   - Round-trip safe (JSON → Binary → JSON)

3. **Excel Export/Import**
   - Export action mappings to Excel
   - User-friendly editing interface
   - Add new mappings
   - Delete existing mappings
   - Import back to JSON

4. **Safety & Validation**
   - Verify integrity before writing to server
   - Backup original configuration
   - Validate against schema
   - Error handling and recovery

## Architecture

### Data Flow

```
┌─────────────────────────────────────────────────────────────┐
│                    GeViServer                                │
│                         ↓                                    │
│              SetupClient API (ReadSetup)                     │
└─────────────────────────────────────────────────────────────┘
                          ↓
                    .set file (binary)
                    281,714 bytes
                          ↓
┌─────────────────────────────────────────────────────────────┐
│                  PHASE 1: Binary Parser                      │
│  - Parse header                                              │
│  - Parse all sections                                        │
│  - Parse all items                                           │
│  - Parse all rules                                           │
│  - Extract action mappings                                   │
└─────────────────────────────────────────────────────────────┘
                          ↓
                    JSON Structure
            (full configuration representation)
                          ↓
┌─────────────────────────────────────────────────────────────┐
│              PHASE 2: JSON Processing                        │
│  - Validate structure                                        │
│  - Transform for editing                                     │
│  - Extract sections                                          │
└─────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────┐
│              PHASE 3: Excel Export                           │
│  - Convert action mappings to Excel                          │
│  - User edits in Excel                                       │
│  - Add/delete/modify mappings                                │
└─────────────────────────────────────────────────────────────┘
                          ↓
                    Excel file
                          ↓
┌─────────────────────────────────────────────────────────────┐
│              PHASE 4: Excel Import                           │
│  - Read Excel changes                                        │
│  - Validate data                                             │
│  - Update JSON structure                                     │
└─────────────────────────────────────────────────────────────┘
                          ↓
                    JSON Structure
                     (modified)
                          ↓
┌─────────────────────────────────────────────────────────────┐
│              PHASE 5: Binary Writer                          │
│  - Rebuild .set file from JSON                               │
│  - Maintain binary format                                    │
│  - Validate integrity                                        │
└─────────────────────────────────────────────────────────────┘
                          ↓
                    .set file (binary)
                          ↓
              SetupClient API (WriteSetup)
                          ↓
                    GeViServer
```

## Binary Format Analysis

### File Structure

```
.set file
├── Header
│   ├── 0x00 (optional null byte)
│   └── Pascal String: "GeViSoft Parameters" (0x07 <len> <data>)
│
├── Sections (multiple)
│   ├── Section Name (Pascal String)
│   ├── Items (key-value pairs)
│   │   ├── Key (Pascal String)
│   │   └── Value (typed)
│   │       ├── 0x01 = Boolean
│   │       ├── 0x04 = Integer (4 bytes)
│   │       └── 0x07 = String (Pascal)
│   │
│   └── Rules Subsection
│       ├── "Rules" marker (0x05 0x52 0x75 0x6C 0x65 0x73)
│       ├── Count/Metadata
│       └── Action Rules (multiple)
│           ├── Trigger Properties
│           │   └── .PropertyName = Boolean
│           ├── Main Action String
│           │   └── 0x07 0x01 0x40 <len_2bytes> <action_data>
│           └── Action Variations
│               ├── GscAction (GeViScope)
│               ├── GNGAction (G-Net-Guard)
│               └── GCoreAction (GCore)
│
└── Footer (metadata/checksums?)
```

### Data Types Discovered

| Marker | Type    | Format                           | Example                    |
|--------|---------|----------------------------------|----------------------------|
| 0x01   | Boolean | 0x01 <value>                     | 0x01 0x01 = true          |
| 0x04   | Integer | 0x04 <4-byte little-endian>      | 0x04 0x0A 0x00 0x00 0x00  |
| 0x07   | String  | 0x07 <len> <data>                | 0x07 0x0B "Description"   |
| 0x07 0x01 0x40 | Action | 0x07 0x01 0x40 <len_2bytes> <data> | Action string format |

### Action String Format

Pattern: `07 01 40 <len_2bytes_LE> <action_text>`

Example:
```
07 01 40 1C 00 47 53 43 20 56 69 65 77 65 72 43 6F 6E 6E 65 63 74 4C 69 76 65...
│  │  │  │  │  └─ "GSC ViewerConnectLive V <- C"
│  │  │  └──┴─ Length: 0x001C (28 bytes)
│  │  └─ 0x40 (action marker)
│  └─ 0x01 (subtype)
└─ 0x07 (string type)
```

### Sections Found

From file analysis, sections include:
- **Alarms**: Alarm configurations
- **Clients**: Client connections
- **GeViIO**: Digital I/O configurations
- **Cameras**: Camera settings
- **Description**: Various descriptive entries
- **IpHost**: Network configurations
- **ActionMappings**: Trigger → Action rules (our focus)

## JSON Schema

### Complete Structure

```json
{
  "$schema": "https://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "version": {
      "type": "string",
      "description": "Parser version"
    },
    "header": {
      "type": "string",
      "description": "File header (GeViSoft Parameters)"
    },
    "sections": {
      "type": "array",
      "items": {
        "$ref": "#/definitions/Section"
      }
    }
  },
  "definitions": {
    "Section": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string"
        },
        "items": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/ConfigItem"
          }
        },
        "rules": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/ActionRule"
          }
        }
      }
    },
    "ConfigItem": {
      "type": "object",
      "properties": {
        "key": {
          "type": "string"
        },
        "value": {
          "oneOf": [
            { "type": "boolean" },
            { "type": "integer" },
            { "type": "string" }
          ]
        },
        "type": {
          "enum": ["boolean", "integer", "string"]
        }
      }
    },
    "ActionRule": {
      "type": "object",
      "properties": {
        "id": {
          "type": "integer"
        },
        "triggers": {
          "type": "object",
          "additionalProperties": {
            "type": "boolean"
          }
        },
        "mainAction": {
          "type": "string"
        },
        "variations": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/ActionVariation"
          }
        }
      }
    },
    "ActionVariation": {
      "type": "object",
      "properties": {
        "platform": {
          "enum": ["GSC", "GNG", "GCore"]
        },
        "actionString": {
          "type": "string"
        },
        "serverType": {
          "type": "string"
        },
        "serverName": {
          "type": "string"
        }
      }
    }
  }
}
```

### Example JSON Output

```json
{
  "version": "1.0",
  "header": "GeViSoft Parameters",
  "sections": [
    {
      "name": "ActionMappings",
      "items": [],
      "rules": [
        {
          "id": 1,
          "triggers": {
            "InputContact": true,
            "VideoInput": false
          },
          "mainAction": "AlternateContact(2, 1000, 500)",
          "variations": [
            {
              "platform": "GSC",
              "actionString": "GSC ViewerConnectLive V <- C_101027",
              "serverType": "GeViScope",
              "serverName": "GEVISCOPE"
            },
            {
              "platform": "GNG",
              "actionString": "GNG PanLeft_101027",
              "serverType": "",
              "serverName": ""
            }
          ]
        }
      ]
    },
    {
      "name": "Alarms",
      "items": [
        {
          "key": "AlarmCount",
          "value": 5,
          "type": "integer"
        },
        {
          "key": "Enabled",
          "value": true,
          "type": "boolean"
        }
      ],
      "rules": []
    }
  ]
}
```

## Implementation Phases

### Phase 1: Complete Binary Parser ✅

**Goal**: Parse entire .set file structure into memory

**Components**:
- ✅ Header parser
- 🚧 Section parser (all types)
- 🚧 Item parser (all data types)
- 🚧 Rules parser (complete structure)
- 🚧 Action variation parser

**Status**: Basic parser exists, needs enhancement for full structure

### Phase 2: JSON Serialization 🚧

**Goal**: Convert parsed structure to JSON

**Components**:
- JSON serializer
- Schema validator
- Round-trip tester (Binary → JSON → Binary)

**Deliverables**:
- `SetFileToJson` converter
- JSON schema definition
- Validation tools

### Phase 3: Excel Export 🚧

**Goal**: Export action mappings to Excel for editing

**Components**:
- Excel writer (EPPlus library)
- Action mapping table generator
- Template with formulas/validation

**Excel Structure**:
```
Sheet: ActionMappings
| Rule ID | Trigger Type | Trigger Param | Action 1 | Action 2 | Action 3 |
|---------|--------------|---------------|----------|----------|----------|
| 1       | InputContact | 3, false      | Alternate| Viewer   |          |
| 2       | VideoInput   | 4, true       | CrossSwi | VCChange |          |
```

### Phase 4: Excel Import 🚧

**Goal**: Import edited Excel back to JSON

**Components**:
- Excel reader
- Validation engine
- Diff generator (show changes)
- JSON merger

### Phase 5: Binary Writer 🚧

**Goal**: Rebuild .set file from JSON

**Components**:
- Binary writer
- Structure rebuilder
- Validation
- Backup mechanism

**Critical**: Must maintain binary compatibility!

### Phase 6: Testing & Validation 🚧

**Goal**: Ensure safety and correctness

**Test Cases**:
1. Round-trip (Binary → JSON → Binary) = identical
2. Round-trip (Binary → JSON → Excel → JSON → Binary) = valid
3. Add new mapping → write → server accepts
4. Modify existing mapping → write → server accepts
5. Delete mapping → write → server accepts

## Current Progress

### Completed ✅

- [x] SetupClient API integration
- [x] Password encryption
- [x] Basic binary parsing (64 action mappings extracted)
- [x] Safe round-trip (byte-for-byte identical)
- [x] File structure analysis
- [x] Data type discovery

### In Progress 🚧

- [ ] Complete section parsing
- [ ] Full rule structure parsing
- [ ] JSON serialization
- [ ] Excel export
- [ ] Binary writer for modifications

### Pending 📋

- [ ] Excel import
- [ ] Add new mapping functionality
- [ ] API endpoints
- [ ] Documentation
- [ ] Production deployment

## Technical Challenges

### Challenge 1: Unknown Metadata Bytes

**Problem**: Many byte sequences whose purpose is unknown

**Solution**:
- Document all patterns found
- Test modifications to understand behavior
- Preserve unknown bytes during round-trip

### Challenge 2: Complex Nested Structure

**Problem**: Sections contain items and rules, rules contain variations

**Solution**:
- Recursive parsing
- Clear data model hierarchy
- Offset tracking for debugging

### Challenge 3: Binary Format Changes

**Problem**: Format may vary between GeViSoft versions

**Solution**:
- Version detection
- Support multiple format versions
- Graceful degradation

### Challenge 4: Action String Syntax

**Problem**: Action strings have complex syntax (parameters, types, etc.)

**Solution**:
- Pattern matching
- Action string parser
- Validation against known action types

## Safety Considerations

### Before Writing to Server

1. ✅ **Verify round-trip**: Parse → Write → Compare = Identical
2. ✅ **Backup original**: Always keep copy of working config
3. ⚠️ **Test in dev**: Never test on production first
4. ⚠️ **Validate structure**: Check against schema
5. ⚠️ **Incremental changes**: Small changes, test frequently

### Error Handling

- Validate before write
- Provide detailed error messages
- Support rollback
- Log all operations

## Tools & Libraries

### Development

- **Language**: C# / .NET 8.0
- **Binary Parsing**: Custom binary reader
- **JSON**: System.Text.Json
- **Excel**: EPPlus (for .xlsx)
- **Testing**: xUnit
- **Logging**: Serilog

### Project Structure

```
GeViSetEditor/
├── GeViSetEditor.Core/
│   ├── Models/
│   │   ├── SetFileStructure.cs
│   │   ├── Section.cs
│   │   ├── ConfigItem.cs
│   │   ├── ActionRule.cs
│   │   └── ActionVariation.cs
│   ├── Parsers/
│   │   ├── SetFileBinaryParser.cs
│   │   ├── SectionParser.cs
│   │   └── RuleParser.cs
│   ├── Writers/
│   │   ├── SetFileBinaryWriter.cs
│   │   └── JsonWriter.cs
│   ├── Converters/
│   │   ├── JsonToExcel.cs
│   │   └── ExcelToJson.cs
│   └── Validators/
│       └── StructureValidator.cs
├── GeViSetEditor.CLI/
│   └── Commands/
│       ├── ParseCommand.cs
│       ├── ToJsonCommand.cs
│       ├── ToExcelCommand.cs
│       └── FromExcelCommand.cs
└── GeViSetEditor.Tests/
    ├── ParserTests.cs
    ├── RoundTripTests.cs
    └── ValidationTests.cs
```

## Next Steps

### Immediate (This Session)

1. ✅ Create specification document
2. ✅ Update git repository
3. 🚧 Implement complete binary parser
4. 🚧 Implement JSON serialization
5. 🚧 Test round-trip with JSON

### Short Term (Next Session)

1. Excel export implementation
2. Excel import implementation
3. Add new mapping functionality
4. Comprehensive testing

### Long Term

1. Web UI for configuration management
2. API endpoints
3. Multi-version support
4. Documentation and examples

## References

- GeViSoft SDK Documentation
- SetupClient API Reference
- Existing .set file samples (TestMKS.set, setup_config_*.dat)
- Binary analysis notes
- Round-trip test results

## Version History

| Version | Date       | Changes                              |
|---------|------------|--------------------------------------|
| 1.0     | 2024-12-12 | Initial specification                |

---

**Status**: Ready for full implementation
**Priority**: High
**Complexity**: High
**Timeline**: 2-3 days estimated
