24a11cecdd
- Add comprehensive spec for .set file format parsing - Document binary structure, data types, and sections - Add research notes from binary analysis - Fix SetupClient password encryption (GeViAPI_EncodeString) - Add DiagnoseSetupClient tool for testing - Successfully tested: read/write 281KB config, byte-perfect round-trip - Found 64 action mappings in live server configuration Next: Full binary parser implementation for complete structure 🤖 Generated with Claude Code Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
618 lines
18 KiB
Markdown
618 lines
18 KiB
Markdown
# GeViSet File Format Reverse Engineering Specification
|
|
|
|
**Version:** 1.0
|
|
**Date:** 2024-12-12
|
|
**Status:** In Progress
|
|
|
|
## Overview
|
|
|
|
This specification documents the reverse engineering effort to fully parse, understand, and manipulate the GeViSoft `.set` configuration file format. The goal is to enable programmatic reading, editing, and writing of GeViServer configurations, particularly action mappings.
|
|
|
|
## Background
|
|
|
|
### What is a .set File?
|
|
|
|
- **Source**: Exported from GeViSet application or read via `GeViAPI_SetupClient_ReadSetup`
|
|
- **Purpose**: Complete GeViServer configuration (cameras, alarms, action mappings, users, etc.)
|
|
- **Format**: Proprietary binary format with "GeViSoft Parameters" header
|
|
- **Size**: Typically 200-300 KB for production configurations
|
|
- **Use Case**: Backup, migration, and programmatic configuration management
|
|
|
|
### Current State
|
|
|
|
**What Works:**
|
|
- ✅ Read .set file from GeViServer via SetupClient API
|
|
- ✅ Extract 64 action mappings from binary data
|
|
- ✅ Write back byte-for-byte identical (round-trip verified)
|
|
- ✅ Password encryption with `GeViAPI_EncodeString`
|
|
|
|
**What's Missing:**
|
|
- ❌ Full structure parsing (all sections, all items)
|
|
- ❌ Understanding of all data types and relationships
|
|
- ❌ Ability to add NEW action mappings
|
|
- ❌ JSON representation of complete structure
|
|
- ❌ Comprehensive Excel export/import
|
|
|
|
## Requirements
|
|
|
|
### Primary Request
|
|
|
|
> "I want to do full reverse engineering - I need to parse the whole file and maybe to json format in the first phase and then we will revert this json or its parts to excel"
|
|
|
|
### Key Requirements
|
|
|
|
1. **Parse Entire File Structure**
|
|
- All sections (Alarms, Clients, GeViIO, Cameras, ActionMappings, etc.)
|
|
- All configuration items (key-value pairs)
|
|
- All rules and triggers
|
|
- All metadata and relationships
|
|
|
|
2. **JSON Serialization**
|
|
- Complete structure in JSON format
|
|
- Human-readable and editable
|
|
- Preserves all data and relationships
|
|
- Round-trip safe (JSON → Binary → JSON)
|
|
|
|
3. **Excel Export/Import**
|
|
- Export action mappings to Excel
|
|
- User-friendly editing interface
|
|
- Add new mappings
|
|
- Delete existing mappings
|
|
- Import back to JSON
|
|
|
|
4. **Safety & Validation**
|
|
- Verify integrity before writing to server
|
|
- Backup original configuration
|
|
- Validate against schema
|
|
- Error handling and recovery
|
|
|
|
## Architecture
|
|
|
|
### Data Flow
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ GeViServer │
|
|
│ ↓ │
|
|
│ SetupClient API (ReadSetup) │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
↓
|
|
.set file (binary)
|
|
281,714 bytes
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ PHASE 1: Binary Parser │
|
|
│ - Parse header │
|
|
│ - Parse all sections │
|
|
│ - Parse all items │
|
|
│ - Parse all rules │
|
|
│ - Extract action mappings │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
↓
|
|
JSON Structure
|
|
(full configuration representation)
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ PHASE 2: JSON Processing │
|
|
│ - Validate structure │
|
|
│ - Transform for editing │
|
|
│ - Extract sections │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ PHASE 3: Excel Export │
|
|
│ - Convert action mappings to Excel │
|
|
│ - User edits in Excel │
|
|
│ - Add/delete/modify mappings │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
↓
|
|
Excel file
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ PHASE 4: Excel Import │
|
|
│ - Read Excel changes │
|
|
│ - Validate data │
|
|
│ - Update JSON structure │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
↓
|
|
JSON Structure
|
|
(modified)
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ PHASE 5: Binary Writer │
|
|
│ - Rebuild .set file from JSON │
|
|
│ - Maintain binary format │
|
|
│ - Validate integrity │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
↓
|
|
.set file (binary)
|
|
↓
|
|
SetupClient API (WriteSetup)
|
|
↓
|
|
GeViServer
|
|
```
|
|
|
|
## Binary Format Analysis
|
|
|
|
### File Structure
|
|
|
|
```
|
|
.set file
|
|
├── Header
|
|
│ ├── 0x00 (optional null byte)
|
|
│ └── Pascal String: "GeViSoft Parameters" (0x07 <len> <data>)
|
|
│
|
|
├── Sections (multiple)
|
|
│ ├── Section Name (Pascal String)
|
|
│ ├── Items (key-value pairs)
|
|
│ │ ├── Key (Pascal String)
|
|
│ │ └── Value (typed)
|
|
│ │ ├── 0x01 = Boolean
|
|
│ │ ├── 0x04 = Integer (4 bytes)
|
|
│ │ └── 0x07 = String (Pascal)
|
|
│ │
|
|
│ └── Rules Subsection
|
|
│ ├── "Rules" marker (0x05 0x52 0x75 0x6C 0x65 0x73)
|
|
│ ├── Count/Metadata
|
|
│ └── Action Rules (multiple)
|
|
│ ├── Trigger Properties
|
|
│ │ └── .PropertyName = Boolean
|
|
│ ├── Main Action String
|
|
│ │ └── 0x07 0x01 0x40 <len_2bytes> <action_data>
|
|
│ └── Action Variations
|
|
│ ├── GscAction (GeViScope)
|
|
│ ├── GNGAction (G-Net-Guard)
|
|
│ └── GCoreAction (GCore)
|
|
│
|
|
└── Footer (metadata/checksums?)
|
|
```
|
|
|
|
### Data Types Discovered
|
|
|
|
| Marker | Type | Format | Example |
|
|
|--------|---------|----------------------------------|----------------------------|
|
|
| 0x01 | Boolean | 0x01 <value> | 0x01 0x01 = true |
|
|
| 0x04 | Integer | 0x04 <4-byte little-endian> | 0x04 0x0A 0x00 0x00 0x00 |
|
|
| 0x07 | String | 0x07 <len> <data> | 0x07 0x0B "Description" |
|
|
| 0x07 0x01 0x40 | Action | 0x07 0x01 0x40 <len_2bytes> <data> | Action string format |
|
|
|
|
### Action String Format
|
|
|
|
Pattern: `07 01 40 <len_2bytes_LE> <action_text>`
|
|
|
|
Example:
|
|
```
|
|
07 01 40 1C 00 47 53 43 20 56 69 65 77 65 72 43 6F 6E 6E 65 63 74 4C 69 76 65...
|
|
│ │ │ │ │ └─ "GSC ViewerConnectLive V <- C"
|
|
│ │ │ └──┴─ Length: 0x001C (28 bytes)
|
|
│ │ └─ 0x40 (action marker)
|
|
│ └─ 0x01 (subtype)
|
|
└─ 0x07 (string type)
|
|
```
|
|
|
|
### Sections Found
|
|
|
|
From file analysis, sections include:
|
|
- **Alarms**: Alarm configurations
|
|
- **Clients**: Client connections
|
|
- **GeViIO**: Digital I/O configurations
|
|
- **Cameras**: Camera settings
|
|
- **Description**: Various descriptive entries
|
|
- **IpHost**: Network configurations
|
|
- **ActionMappings**: Trigger → Action rules (our focus)
|
|
|
|
## JSON Schema
|
|
|
|
### Complete Structure
|
|
|
|
```json
|
|
{
|
|
"$schema": "https://json-schema.org/draft-07/schema#",
|
|
"type": "object",
|
|
"properties": {
|
|
"version": {
|
|
"type": "string",
|
|
"description": "Parser version"
|
|
},
|
|
"header": {
|
|
"type": "string",
|
|
"description": "File header (GeViSoft Parameters)"
|
|
},
|
|
"sections": {
|
|
"type": "array",
|
|
"items": {
|
|
"$ref": "#/definitions/Section"
|
|
}
|
|
}
|
|
},
|
|
"definitions": {
|
|
"Section": {
|
|
"type": "object",
|
|
"properties": {
|
|
"name": {
|
|
"type": "string"
|
|
},
|
|
"items": {
|
|
"type": "array",
|
|
"items": {
|
|
"$ref": "#/definitions/ConfigItem"
|
|
}
|
|
},
|
|
"rules": {
|
|
"type": "array",
|
|
"items": {
|
|
"$ref": "#/definitions/ActionRule"
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"ConfigItem": {
|
|
"type": "object",
|
|
"properties": {
|
|
"key": {
|
|
"type": "string"
|
|
},
|
|
"value": {
|
|
"oneOf": [
|
|
{ "type": "boolean" },
|
|
{ "type": "integer" },
|
|
{ "type": "string" }
|
|
]
|
|
},
|
|
"type": {
|
|
"enum": ["boolean", "integer", "string"]
|
|
}
|
|
}
|
|
},
|
|
"ActionRule": {
|
|
"type": "object",
|
|
"properties": {
|
|
"id": {
|
|
"type": "integer"
|
|
},
|
|
"triggers": {
|
|
"type": "object",
|
|
"additionalProperties": {
|
|
"type": "boolean"
|
|
}
|
|
},
|
|
"mainAction": {
|
|
"type": "string"
|
|
},
|
|
"variations": {
|
|
"type": "array",
|
|
"items": {
|
|
"$ref": "#/definitions/ActionVariation"
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"ActionVariation": {
|
|
"type": "object",
|
|
"properties": {
|
|
"platform": {
|
|
"enum": ["GSC", "GNG", "GCore"]
|
|
},
|
|
"actionString": {
|
|
"type": "string"
|
|
},
|
|
"serverType": {
|
|
"type": "string"
|
|
},
|
|
"serverName": {
|
|
"type": "string"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Example JSON Output
|
|
|
|
```json
|
|
{
|
|
"version": "1.0",
|
|
"header": "GeViSoft Parameters",
|
|
"sections": [
|
|
{
|
|
"name": "ActionMappings",
|
|
"items": [],
|
|
"rules": [
|
|
{
|
|
"id": 1,
|
|
"triggers": {
|
|
"InputContact": true,
|
|
"VideoInput": false
|
|
},
|
|
"mainAction": "AlternateContact(2, 1000, 500)",
|
|
"variations": [
|
|
{
|
|
"platform": "GSC",
|
|
"actionString": "GSC ViewerConnectLive V <- C_101027",
|
|
"serverType": "GeViScope",
|
|
"serverName": "GEVISCOPE"
|
|
},
|
|
{
|
|
"platform": "GNG",
|
|
"actionString": "GNG PanLeft_101027",
|
|
"serverType": "",
|
|
"serverName": ""
|
|
}
|
|
]
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"name": "Alarms",
|
|
"items": [
|
|
{
|
|
"key": "AlarmCount",
|
|
"value": 5,
|
|
"type": "integer"
|
|
},
|
|
{
|
|
"key": "Enabled",
|
|
"value": true,
|
|
"type": "boolean"
|
|
}
|
|
],
|
|
"rules": []
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## Implementation Phases
|
|
|
|
### Phase 1: Complete Binary Parser ✅
|
|
|
|
**Goal**: Parse entire .set file structure into memory
|
|
|
|
**Components**:
|
|
- ✅ Header parser
|
|
- 🚧 Section parser (all types)
|
|
- 🚧 Item parser (all data types)
|
|
- 🚧 Rules parser (complete structure)
|
|
- 🚧 Action variation parser
|
|
|
|
**Status**: Basic parser exists, needs enhancement for full structure
|
|
|
|
### Phase 2: JSON Serialization 🚧
|
|
|
|
**Goal**: Convert parsed structure to JSON
|
|
|
|
**Components**:
|
|
- JSON serializer
|
|
- Schema validator
|
|
- Round-trip tester (Binary → JSON → Binary)
|
|
|
|
**Deliverables**:
|
|
- `SetFileToJson` converter
|
|
- JSON schema definition
|
|
- Validation tools
|
|
|
|
### Phase 3: Excel Export 🚧
|
|
|
|
**Goal**: Export action mappings to Excel for editing
|
|
|
|
**Components**:
|
|
- Excel writer (EPPlus library)
|
|
- Action mapping table generator
|
|
- Template with formulas/validation
|
|
|
|
**Excel Structure**:
|
|
```
|
|
Sheet: ActionMappings
|
|
| Rule ID | Trigger Type | Trigger Param | Action 1 | Action 2 | Action 3 |
|
|
|---------|--------------|---------------|----------|----------|----------|
|
|
| 1 | InputContact | 3, false | Alternate| Viewer | |
|
|
| 2 | VideoInput | 4, true | CrossSwi | VCChange | |
|
|
```
|
|
|
|
### Phase 4: Excel Import 🚧
|
|
|
|
**Goal**: Import edited Excel back to JSON
|
|
|
|
**Components**:
|
|
- Excel reader
|
|
- Validation engine
|
|
- Diff generator (show changes)
|
|
- JSON merger
|
|
|
|
### Phase 5: Binary Writer 🚧
|
|
|
|
**Goal**: Rebuild .set file from JSON
|
|
|
|
**Components**:
|
|
- Binary writer
|
|
- Structure rebuilder
|
|
- Validation
|
|
- Backup mechanism
|
|
|
|
**Critical**: Must maintain binary compatibility!
|
|
|
|
### Phase 6: Testing & Validation 🚧
|
|
|
|
**Goal**: Ensure safety and correctness
|
|
|
|
**Test Cases**:
|
|
1. Round-trip (Binary → JSON → Binary) = identical
|
|
2. Round-trip (Binary → JSON → Excel → JSON → Binary) = valid
|
|
3. Add new mapping → write → server accepts
|
|
4. Modify existing mapping → write → server accepts
|
|
5. Delete mapping → write → server accepts
|
|
|
|
## Current Progress
|
|
|
|
### Completed ✅
|
|
|
|
- [x] SetupClient API integration
|
|
- [x] Password encryption
|
|
- [x] Basic binary parsing (64 action mappings extracted)
|
|
- [x] Safe round-trip (byte-for-byte identical)
|
|
- [x] File structure analysis
|
|
- [x] Data type discovery
|
|
|
|
### In Progress 🚧
|
|
|
|
- [ ] Complete section parsing
|
|
- [ ] Full rule structure parsing
|
|
- [ ] JSON serialization
|
|
- [ ] Excel export
|
|
- [ ] Binary writer for modifications
|
|
|
|
### Pending 📋
|
|
|
|
- [ ] Excel import
|
|
- [ ] Add new mapping functionality
|
|
- [ ] API endpoints
|
|
- [ ] Documentation
|
|
- [ ] Production deployment
|
|
|
|
## Technical Challenges
|
|
|
|
### Challenge 1: Unknown Metadata Bytes
|
|
|
|
**Problem**: Many byte sequences whose purpose is unknown
|
|
|
|
**Solution**:
|
|
- Document all patterns found
|
|
- Test modifications to understand behavior
|
|
- Preserve unknown bytes during round-trip
|
|
|
|
### Challenge 2: Complex Nested Structure
|
|
|
|
**Problem**: Sections contain items and rules, rules contain variations
|
|
|
|
**Solution**:
|
|
- Recursive parsing
|
|
- Clear data model hierarchy
|
|
- Offset tracking for debugging
|
|
|
|
### Challenge 3: Binary Format Changes
|
|
|
|
**Problem**: Format may vary between GeViSoft versions
|
|
|
|
**Solution**:
|
|
- Version detection
|
|
- Support multiple format versions
|
|
- Graceful degradation
|
|
|
|
### Challenge 4: Action String Syntax
|
|
|
|
**Problem**: Action strings have complex syntax (parameters, types, etc.)
|
|
|
|
**Solution**:
|
|
- Pattern matching
|
|
- Action string parser
|
|
- Validation against known action types
|
|
|
|
## Safety Considerations
|
|
|
|
### Before Writing to Server
|
|
|
|
1. ✅ **Verify round-trip**: Parse → Write → Compare = Identical
|
|
2. ✅ **Backup original**: Always keep copy of working config
|
|
3. ⚠️ **Test in dev**: Never test on production first
|
|
4. ⚠️ **Validate structure**: Check against schema
|
|
5. ⚠️ **Incremental changes**: Small changes, test frequently
|
|
|
|
### Error Handling
|
|
|
|
- Validate before write
|
|
- Provide detailed error messages
|
|
- Support rollback
|
|
- Log all operations
|
|
|
|
## Tools & Libraries
|
|
|
|
### Development
|
|
|
|
- **Language**: C# / .NET 8.0
|
|
- **Binary Parsing**: Custom binary reader
|
|
- **JSON**: System.Text.Json
|
|
- **Excel**: EPPlus (for .xlsx)
|
|
- **Testing**: xUnit
|
|
- **Logging**: Serilog
|
|
|
|
### Project Structure
|
|
|
|
```
|
|
GeViSetEditor/
|
|
├── GeViSetEditor.Core/
|
|
│ ├── Models/
|
|
│ │ ├── SetFileStructure.cs
|
|
│ │ ├── Section.cs
|
|
│ │ ├── ConfigItem.cs
|
|
│ │ ├── ActionRule.cs
|
|
│ │ └── ActionVariation.cs
|
|
│ ├── Parsers/
|
|
│ │ ├── SetFileBinaryParser.cs
|
|
│ │ ├── SectionParser.cs
|
|
│ │ └── RuleParser.cs
|
|
│ ├── Writers/
|
|
│ │ ├── SetFileBinaryWriter.cs
|
|
│ │ └── JsonWriter.cs
|
|
│ ├── Converters/
|
|
│ │ ├── JsonToExcel.cs
|
|
│ │ └── ExcelToJson.cs
|
|
│ └── Validators/
|
|
│ └── StructureValidator.cs
|
|
├── GeViSetEditor.CLI/
|
|
│ └── Commands/
|
|
│ ├── ParseCommand.cs
|
|
│ ├── ToJsonCommand.cs
|
|
│ ├── ToExcelCommand.cs
|
|
│ └── FromExcelCommand.cs
|
|
└── GeViSetEditor.Tests/
|
|
├── ParserTests.cs
|
|
├── RoundTripTests.cs
|
|
└── ValidationTests.cs
|
|
```
|
|
|
|
## Next Steps
|
|
|
|
### Immediate (This Session)
|
|
|
|
1. ✅ Create specification document
|
|
2. ✅ Update git repository
|
|
3. 🚧 Implement complete binary parser
|
|
4. 🚧 Implement JSON serialization
|
|
5. 🚧 Test round-trip with JSON
|
|
|
|
### Short Term (Next Session)
|
|
|
|
1. Excel export implementation
|
|
2. Excel import implementation
|
|
3. Add new mapping functionality
|
|
4. Comprehensive testing
|
|
|
|
### Long Term
|
|
|
|
1. Web UI for configuration management
|
|
2. API endpoints
|
|
3. Multi-version support
|
|
4. Documentation and examples
|
|
|
|
## References
|
|
|
|
- GeViSoft SDK Documentation
|
|
- SetupClient API Reference
|
|
- Existing .set file samples (TestMKS.set, setup_config_*.dat)
|
|
- Binary analysis notes
|
|
- Round-trip test results
|
|
|
|
## Version History
|
|
|
|
| Version | Date | Changes |
|
|
|---------|------------|--------------------------------------|
|
|
| 1.0 | 2024-12-12 | Initial specification |
|
|
|
|
---
|
|
|
|
**Status**: Ready for full implementation
|
|
**Priority**: High
|
|
**Complexity**: High
|
|
**Timeline**: 2-3 days estimated
|