feat: Add GeViSet file format reverse engineering specification
- Add comprehensive spec for .set file format parsing - Document binary structure, data types, and sections - Add research notes from binary analysis - Fix SetupClient password encryption (GeViAPI_EncodeString) - Add DiagnoseSetupClient tool for testing - Successfully tested: read/write 281KB config, byte-perfect round-trip - Found 64 action mappings in live server configuration Next: Full binary parser implementation for complete structure 🤖 Generated with Claude Code Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Geutebruck API Developer
617
specs/002-geviset-file-format/spec.md
Normal file
617
specs/002-geviset-file-format/spec.md
Normal file
@@ -0,0 +1,617 @@
|
||||
# GeViSet File Format Reverse Engineering Specification
|
||||
|
||||
**Version:** 1.0
|
||||
**Date:** 2024-12-12
|
||||
**Status:** In Progress
|
||||
|
||||
## Overview
|
||||
|
||||
This specification documents the reverse engineering effort to fully parse, understand, and manipulate the GeViSoft `.set` configuration file format. The goal is to enable programmatic reading, editing, and writing of GeViServer configurations, particularly action mappings.
|
||||
|
||||
## Background
|
||||
|
||||
### What is a .set File?
|
||||
|
||||
- **Source**: Exported from GeViSet application or read via `GeViAPI_SetupClient_ReadSetup`
|
||||
- **Purpose**: Complete GeViServer configuration (cameras, alarms, action mappings, users, etc.)
|
||||
- **Format**: Proprietary binary format with "GeViSoft Parameters" header
|
||||
- **Size**: Typically 200-300 KB for production configurations
|
||||
- **Use Case**: Backup, migration, and programmatic configuration management
|
||||
|
||||
### Current State
|
||||
|
||||
**What Works:**
|
||||
- ✅ Read .set file from GeViServer via SetupClient API
|
||||
- ✅ Extract 64 action mappings from binary data
|
||||
- ✅ Write back byte-for-byte identical (round-trip verified)
|
||||
- ✅ Password encryption with `GeViAPI_EncodeString`
|
||||
|
||||
**What's Missing:**
|
||||
- ❌ Full structure parsing (all sections, all items)
|
||||
- ❌ Understanding of all data types and relationships
|
||||
- ❌ Ability to add NEW action mappings
|
||||
- ❌ JSON representation of complete structure
|
||||
- ❌ Comprehensive Excel export/import
|
||||
|
||||
## Requirements
|
||||
|
||||
### Primary Request
|
||||
|
||||
> "I want to do full reverse engineering - I need to parse the whole file and maybe to json format in the first phase and then we will revert this json or its parts to excel"
|
||||
|
||||
### Key Requirements
|
||||
|
||||
1. **Parse Entire File Structure**
|
||||
- All sections (Alarms, Clients, GeViIO, Cameras, ActionMappings, etc.)
|
||||
- All configuration items (key-value pairs)
|
||||
- All rules and triggers
|
||||
- All metadata and relationships
|
||||
|
||||
2. **JSON Serialization**
|
||||
- Complete structure in JSON format
|
||||
- Human-readable and editable
|
||||
- Preserves all data and relationships
|
||||
- Round-trip safe (JSON → Binary → JSON)
|
||||
|
||||
3. **Excel Export/Import**
|
||||
- Export action mappings to Excel
|
||||
- User-friendly editing interface
|
||||
- Add new mappings
|
||||
- Delete existing mappings
|
||||
- Import back to JSON
|
||||
|
||||
4. **Safety & Validation**
|
||||
- Verify integrity before writing to server
|
||||
- Backup original configuration
|
||||
- Validate against schema
|
||||
- Error handling and recovery
|
||||
|
||||
## Architecture
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ GeViServer │
|
||||
│ ↓ │
|
||||
│ SetupClient API (ReadSetup) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
.set file (binary)
|
||||
281,714 bytes
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ PHASE 1: Binary Parser │
|
||||
│ - Parse header │
|
||||
│ - Parse all sections │
|
||||
│ - Parse all items │
|
||||
│ - Parse all rules │
|
||||
│ - Extract action mappings │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
JSON Structure
|
||||
(full configuration representation)
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ PHASE 2: JSON Processing │
|
||||
│ - Validate structure │
|
||||
│ - Transform for editing │
|
||||
│ - Extract sections │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ PHASE 3: Excel Export │
|
||||
│ - Convert action mappings to Excel │
|
||||
│ - User edits in Excel │
|
||||
│ - Add/delete/modify mappings │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
Excel file
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ PHASE 4: Excel Import │
|
||||
│ - Read Excel changes │
|
||||
│ - Validate data │
|
||||
│ - Update JSON structure │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
JSON Structure
|
||||
(modified)
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ PHASE 5: Binary Writer │
|
||||
│ - Rebuild .set file from JSON │
|
||||
│ - Maintain binary format │
|
||||
│ - Validate integrity │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
.set file (binary)
|
||||
↓
|
||||
SetupClient API (WriteSetup)
|
||||
↓
|
||||
GeViServer
|
||||
```
|
||||
|
||||
## Binary Format Analysis
|
||||
|
||||
### File Structure
|
||||
|
||||
```
|
||||
.set file
|
||||
├── Header
|
||||
│ ├── 0x00 (optional null byte)
|
||||
│ └── Pascal String: "GeViSoft Parameters" (0x07 <len> <data>)
|
||||
│
|
||||
├── Sections (multiple)
|
||||
│ ├── Section Name (Pascal String)
|
||||
│ ├── Items (key-value pairs)
|
||||
│ │ ├── Key (Pascal String)
|
||||
│ │ └── Value (typed)
|
||||
│ │ ├── 0x01 = Boolean
|
||||
│ │ ├── 0x04 = Integer (4 bytes)
|
||||
│ │ └── 0x07 = String (Pascal)
|
||||
│ │
|
||||
│ └── Rules Subsection
|
||||
│ ├── "Rules" marker (0x05 0x52 0x75 0x6C 0x65 0x73)
|
||||
│ ├── Count/Metadata
|
||||
│ └── Action Rules (multiple)
|
||||
│ ├── Trigger Properties
|
||||
│ │ └── .PropertyName = Boolean
|
||||
│ ├── Main Action String
|
||||
│ │ └── 0x07 0x01 0x40 <len_2bytes> <action_data>
|
||||
│ └── Action Variations
|
||||
│ ├── GscAction (GeViScope)
|
||||
│ ├── GNGAction (G-Net-Guard)
|
||||
│ └── GCoreAction (GCore)
|
||||
│
|
||||
└── Footer (metadata/checksums?)
|
||||
```
|
||||
|
||||
### Data Types Discovered
|
||||
|
||||
| Marker | Type | Format | Example |
|
||||
|--------|---------|----------------------------------|----------------------------|
|
||||
| 0x01 | Boolean | 0x01 <value> | 0x01 0x01 = true |
|
||||
| 0x04 | Integer | 0x04 <4-byte little-endian> | 0x04 0x0A 0x00 0x00 0x00 |
|
||||
| 0x07 | String | 0x07 <len> <data> | 0x07 0x0B "Description" |
|
||||
| 0x07 0x01 0x40 | Action | 0x07 0x01 0x40 <len_2bytes> <data> | Action string format |
|
||||
|
||||
### Action String Format
|
||||
|
||||
Pattern: `07 01 40 <len_2bytes_LE> <action_text>`
|
||||
|
||||
Example:
|
||||
```
|
||||
07 01 40 1C 00 47 53 43 20 56 69 65 77 65 72 43 6F 6E 6E 65 63 74 4C 69 76 65...
|
||||
│ │ │ │ │ └─ "GSC ViewerConnectLive V <- C"
|
||||
│ │ │ └──┴─ Length: 0x001C (28 bytes)
|
||||
│ │ └─ 0x40 (action marker)
|
||||
│ └─ 0x01 (subtype)
|
||||
└─ 0x07 (string type)
|
||||
```
|
||||
|
||||
### Sections Found
|
||||
|
||||
From file analysis, sections include:
|
||||
- **Alarms**: Alarm configurations
|
||||
- **Clients**: Client connections
|
||||
- **GeViIO**: Digital I/O configurations
|
||||
- **Cameras**: Camera settings
|
||||
- **Description**: Various descriptive entries
|
||||
- **IpHost**: Network configurations
|
||||
- **ActionMappings**: Trigger → Action rules (our focus)
|
||||
|
||||
## JSON Schema
|
||||
|
||||
### Complete Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft-07/schema#",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"version": {
|
||||
"type": "string",
|
||||
"description": "Parser version"
|
||||
},
|
||||
"header": {
|
||||
"type": "string",
|
||||
"description": "File header (GeViSoft Parameters)"
|
||||
},
|
||||
"sections": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/definitions/Section"
|
||||
}
|
||||
}
|
||||
},
|
||||
"definitions": {
|
||||
"Section": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"items": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/definitions/ConfigItem"
|
||||
}
|
||||
},
|
||||
"rules": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/definitions/ActionRule"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"ConfigItem": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"key": {
|
||||
"type": "string"
|
||||
},
|
||||
"value": {
|
||||
"oneOf": [
|
||||
{ "type": "boolean" },
|
||||
{ "type": "integer" },
|
||||
{ "type": "string" }
|
||||
]
|
||||
},
|
||||
"type": {
|
||||
"enum": ["boolean", "integer", "string"]
|
||||
}
|
||||
}
|
||||
},
|
||||
"ActionRule": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "integer"
|
||||
},
|
||||
"triggers": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"mainAction": {
|
||||
"type": "string"
|
||||
},
|
||||
"variations": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/definitions/ActionVariation"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"ActionVariation": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"platform": {
|
||||
"enum": ["GSC", "GNG", "GCore"]
|
||||
},
|
||||
"actionString": {
|
||||
"type": "string"
|
||||
},
|
||||
"serverType": {
|
||||
"type": "string"
|
||||
},
|
||||
"serverName": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Example JSON Output
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "1.0",
|
||||
"header": "GeViSoft Parameters",
|
||||
"sections": [
|
||||
{
|
||||
"name": "ActionMappings",
|
||||
"items": [],
|
||||
"rules": [
|
||||
{
|
||||
"id": 1,
|
||||
"triggers": {
|
||||
"InputContact": true,
|
||||
"VideoInput": false
|
||||
},
|
||||
"mainAction": "AlternateContact(2, 1000, 500)",
|
||||
"variations": [
|
||||
{
|
||||
"platform": "GSC",
|
||||
"actionString": "GSC ViewerConnectLive V <- C_101027",
|
||||
"serverType": "GeViScope",
|
||||
"serverName": "GEVISCOPE"
|
||||
},
|
||||
{
|
||||
"platform": "GNG",
|
||||
"actionString": "GNG PanLeft_101027",
|
||||
"serverType": "",
|
||||
"serverName": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Alarms",
|
||||
"items": [
|
||||
{
|
||||
"key": "AlarmCount",
|
||||
"value": 5,
|
||||
"type": "integer"
|
||||
},
|
||||
{
|
||||
"key": "Enabled",
|
||||
"value": true,
|
||||
"type": "boolean"
|
||||
}
|
||||
],
|
||||
"rules": []
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Complete Binary Parser ✅
|
||||
|
||||
**Goal**: Parse entire .set file structure into memory
|
||||
|
||||
**Components**:
|
||||
- ✅ Header parser
|
||||
- 🚧 Section parser (all types)
|
||||
- 🚧 Item parser (all data types)
|
||||
- 🚧 Rules parser (complete structure)
|
||||
- 🚧 Action variation parser
|
||||
|
||||
**Status**: Basic parser exists, needs enhancement for full structure
|
||||
|
||||
### Phase 2: JSON Serialization 🚧
|
||||
|
||||
**Goal**: Convert parsed structure to JSON
|
||||
|
||||
**Components**:
|
||||
- JSON serializer
|
||||
- Schema validator
|
||||
- Round-trip tester (Binary → JSON → Binary)
|
||||
|
||||
**Deliverables**:
|
||||
- `SetFileToJson` converter
|
||||
- JSON schema definition
|
||||
- Validation tools
|
||||
|
||||
### Phase 3: Excel Export 🚧
|
||||
|
||||
**Goal**: Export action mappings to Excel for editing
|
||||
|
||||
**Components**:
|
||||
- Excel writer (EPPlus library)
|
||||
- Action mapping table generator
|
||||
- Template with formulas/validation
|
||||
|
||||
**Excel Structure**:
|
||||
```
|
||||
Sheet: ActionMappings
|
||||
| Rule ID | Trigger Type | Trigger Param | Action 1 | Action 2 | Action 3 |
|
||||
|---------|--------------|---------------|----------|----------|----------|
|
||||
| 1 | InputContact | 3, false | Alternate| Viewer | |
|
||||
| 2 | VideoInput | 4, true | CrossSwi | VCChange | |
|
||||
```
|
||||
|
||||
### Phase 4: Excel Import 🚧
|
||||
|
||||
**Goal**: Import edited Excel back to JSON
|
||||
|
||||
**Components**:
|
||||
- Excel reader
|
||||
- Validation engine
|
||||
- Diff generator (show changes)
|
||||
- JSON merger
|
||||
|
||||
### Phase 5: Binary Writer 🚧
|
||||
|
||||
**Goal**: Rebuild .set file from JSON
|
||||
|
||||
**Components**:
|
||||
- Binary writer
|
||||
- Structure rebuilder
|
||||
- Validation
|
||||
- Backup mechanism
|
||||
|
||||
**Critical**: Must maintain binary compatibility!
|
||||
|
||||
### Phase 6: Testing & Validation 🚧
|
||||
|
||||
**Goal**: Ensure safety and correctness
|
||||
|
||||
**Test Cases**:
|
||||
1. Round-trip (Binary → JSON → Binary) = identical
|
||||
2. Round-trip (Binary → JSON → Excel → JSON → Binary) = valid
|
||||
3. Add new mapping → write → server accepts
|
||||
4. Modify existing mapping → write → server accepts
|
||||
5. Delete mapping → write → server accepts
|
||||
|
||||
## Current Progress
|
||||
|
||||
### Completed ✅
|
||||
|
||||
- [x] SetupClient API integration
|
||||
- [x] Password encryption
|
||||
- [x] Basic binary parsing (64 action mappings extracted)
|
||||
- [x] Safe round-trip (byte-for-byte identical)
|
||||
- [x] File structure analysis
|
||||
- [x] Data type discovery
|
||||
|
||||
### In Progress 🚧
|
||||
|
||||
- [ ] Complete section parsing
|
||||
- [ ] Full rule structure parsing
|
||||
- [ ] JSON serialization
|
||||
- [ ] Excel export
|
||||
- [ ] Binary writer for modifications
|
||||
|
||||
### Pending 📋
|
||||
|
||||
- [ ] Excel import
|
||||
- [ ] Add new mapping functionality
|
||||
- [ ] API endpoints
|
||||
- [ ] Documentation
|
||||
- [ ] Production deployment
|
||||
|
||||
## Technical Challenges
|
||||
|
||||
### Challenge 1: Unknown Metadata Bytes
|
||||
|
||||
**Problem**: Many byte sequences whose purpose is unknown
|
||||
|
||||
**Solution**:
|
||||
- Document all patterns found
|
||||
- Test modifications to understand behavior
|
||||
- Preserve unknown bytes during round-trip
|
||||
|
||||
### Challenge 2: Complex Nested Structure
|
||||
|
||||
**Problem**: Sections contain items and rules, rules contain variations
|
||||
|
||||
**Solution**:
|
||||
- Recursive parsing
|
||||
- Clear data model hierarchy
|
||||
- Offset tracking for debugging
|
||||
|
||||
### Challenge 3: Binary Format Changes
|
||||
|
||||
**Problem**: Format may vary between GeViSoft versions
|
||||
|
||||
**Solution**:
|
||||
- Version detection
|
||||
- Support multiple format versions
|
||||
- Graceful degradation
|
||||
|
||||
### Challenge 4: Action String Syntax
|
||||
|
||||
**Problem**: Action strings have complex syntax (parameters, types, etc.)
|
||||
|
||||
**Solution**:
|
||||
- Pattern matching
|
||||
- Action string parser
|
||||
- Validation against known action types
|
||||
|
||||
## Safety Considerations
|
||||
|
||||
### Before Writing to Server
|
||||
|
||||
1. ✅ **Verify round-trip**: Parse → Write → Compare = Identical
|
||||
2. ✅ **Backup original**: Always keep copy of working config
|
||||
3. ⚠️ **Test in dev**: Never test on production first
|
||||
4. ⚠️ **Validate structure**: Check against schema
|
||||
5. ⚠️ **Incremental changes**: Small changes, test frequently
|
||||
|
||||
### Error Handling
|
||||
|
||||
- Validate before write
|
||||
- Provide detailed error messages
|
||||
- Support rollback
|
||||
- Log all operations
|
||||
|
||||
## Tools & Libraries
|
||||
|
||||
### Development
|
||||
|
||||
- **Language**: C# / .NET 8.0
|
||||
- **Binary Parsing**: Custom binary reader
|
||||
- **JSON**: System.Text.Json
|
||||
- **Excel**: EPPlus (for .xlsx)
|
||||
- **Testing**: xUnit
|
||||
- **Logging**: Serilog
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
GeViSetEditor/
|
||||
├── GeViSetEditor.Core/
|
||||
│ ├── Models/
|
||||
│ │ ├── SetFileStructure.cs
|
||||
│ │ ├── Section.cs
|
||||
│ │ ├── ConfigItem.cs
|
||||
│ │ ├── ActionRule.cs
|
||||
│ │ └── ActionVariation.cs
|
||||
│ ├── Parsers/
|
||||
│ │ ├── SetFileBinaryParser.cs
|
||||
│ │ ├── SectionParser.cs
|
||||
│ │ └── RuleParser.cs
|
||||
│ ├── Writers/
|
||||
│ │ ├── SetFileBinaryWriter.cs
|
||||
│ │ └── JsonWriter.cs
|
||||
│ ├── Converters/
|
||||
│ │ ├── JsonToExcel.cs
|
||||
│ │ └── ExcelToJson.cs
|
||||
│ └── Validators/
|
||||
│ └── StructureValidator.cs
|
||||
├── GeViSetEditor.CLI/
|
||||
│ └── Commands/
|
||||
│ ├── ParseCommand.cs
|
||||
│ ├── ToJsonCommand.cs
|
||||
│ ├── ToExcelCommand.cs
|
||||
│ └── FromExcelCommand.cs
|
||||
└── GeViSetEditor.Tests/
|
||||
├── ParserTests.cs
|
||||
├── RoundTripTests.cs
|
||||
└── ValidationTests.cs
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
### Immediate (This Session)
|
||||
|
||||
1. ✅ Create specification document
|
||||
2. ✅ Update git repository
|
||||
3. 🚧 Implement complete binary parser
|
||||
4. 🚧 Implement JSON serialization
|
||||
5. 🚧 Test round-trip with JSON
|
||||
|
||||
### Short Term (Next Session)
|
||||
|
||||
1. Excel export implementation
|
||||
2. Excel import implementation
|
||||
3. Add new mapping functionality
|
||||
4. Comprehensive testing
|
||||
|
||||
### Long Term
|
||||
|
||||
1. Web UI for configuration management
|
||||
2. API endpoints
|
||||
3. Multi-version support
|
||||
4. Documentation and examples
|
||||
|
||||
## References
|
||||
|
||||
- GeViSoft SDK Documentation
|
||||
- SetupClient API Reference
|
||||
- Existing .set file samples (TestMKS.set, setup_config_*.dat)
|
||||
- Binary analysis notes
|
||||
- Round-trip test results
|
||||
|
||||
## Version History
|
||||
|
||||
| Version | Date | Changes |
|
||||
|---------|------------|--------------------------------------|
|
||||
| 1.0 | 2024-12-12 | Initial specification |
|
||||
|
||||
---
|
||||
|
||||
**Status**: Ready for full implementation
|
||||
**Priority**: High
|
||||
**Complexity**: High
|
||||
**Timeline**: 2-3 days estimated
|
||||
Reference in New Issue
Block a user