diff --git a/README.md b/README.md index d7ef4ad..15f3669 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,6 @@ A high-performance, scalable OPC UA data ingestion system designed for industria ### Core Functionality - **OPC UA Integration**: Native support for OPC UA servers with secure and unsecured connections - **Automatic Load Balancing**: Slot-based architecture for horizontal scaling across multiple instances -- **Real-time Data Streaming**: Kafka integration for historical data streaming - **Persistent Storage**: MongoDB integration for historical data persistence - **Health Monitoring**: Comprehensive Prometheus metrics and health checks - **Fault Tolerance**: Automatic failover, reconnection, and error recovery @@ -36,9 +35,9 @@ The OPC Ingestor uses a modular, manager-based architecture designed for scalabi ┌──────────────────┐ ┌─────────────────┐ │ ResourceManager │ │ DataManager │ │ │ │ │ - │ - Redis Coord. │ │ - Kafka Export │ - │ - Slot Leasing │ │ - MongoDB Store │ - │ - Heartbeats │ │ - Data Pipeline │ + │ - Redis Coord. │ │ - MongoDB Store │ + │ - Slot Leasing │ │ - Data Pipeline │ + │ - Heartbeats │ │ │ └──────────────────┘ └─────────────────┘ ``` @@ -47,7 +46,7 @@ The OPC Ingestor uses a modular, manager-based architecture designed for scalabi - **Ingestor**: Main application orchestrator managing the overall lifecycle - **IngestorManager**: Coordinates slot allocation, OPC server management, and load balancing - **OPC Manager**: Handles individual OPC UA server connections and tag subscriptions -- **Data Manager**: Manages data streaming (MongoDB | Kafka) +- **Data Manager**: Manages data persistence (MongoDB) - **Resource Manager**: Coordinates resource allocation and instance coordination via Redis ## 📋 Prerequisites @@ -55,10 +54,9 @@ The OPC Ingestor uses a modular, manager-based architecture designed for scalabi - Python 3.11+ - Redis server - MongoDB server -- Kafka cluster (optional, for data streaming) - OPC UA servers for data collection -**Note**: External dependencies (Redis, MongoDB, Kafka) must be available either through: +**Note**: External dependencies (Redis, MongoDB) must be available either through: - Port forwarding from a Kubernetes cluster - External Docker Compose setup - Cloud-managed services @@ -99,12 +97,15 @@ The OPC Ingestor uses a modular, manager-based architecture designed for scalabi # Port forwarding from Kubernetes cluster kubectl port-forward svc/redis-master 6379:6379 kubectl port-forward svc/mongodb 27017:27017 - kubectl port-forward svc/kafka 9092:9092 # Or connect to external Docker Compose # Ensure services are accessible on localhost with appropriate ports ``` +6. **Generate an OPC UA client certificate** (only if connecting to a secured OPC UA server) + + See [Certificate Generation](#-certificate-generation) below. + ## Usage ### Running the Ingestor @@ -171,6 +172,38 @@ pytest --cov=ingestor --cov-report=html The feeder creates sample OPC tag configurations in Redis that the ingestor can discover and manage. +## 🔐 Certificate Generation + +OPC UA servers that require secure connections (e.g. KEPServer with `Basic256` policy) validate the ingestor via an X.509 client certificate. Use `scripts/generate-and-push-opc-cert.sh` to generate that certificate and push it to the Gitea repo the pod clones on boot. + +```bash +GITEA_PASSWORD='' ./scripts/generate-and-push-opc-cert.sh +``` + +Environment variables (all optional except `GITEA_PASSWORD`): + +| Variable | Description | Default | +|----------|-------------|---------| +| `GITEA_URL` | Gitea in-cluster URL | `https://git.sientia.ai` | +| `GITEA_USER` | Gitea user | `gitea_admin` | +| `GITEA_PASSWORD` | Gitea password (**required**) | — | +| `GITEA_REPO` | Repo the ingestor pod clones | `gitea_admin/sientia-dataops-opc-ingestor` | +| `BRANCH` | Branch to push the cert to | `main` | +| `APP_URI` | SAN URI of the cert — **must match** the `uri` field of the corresponding document in the `OPC_servers` Mongo collection, since the ingestor uses it as `application_uri` (KEPServer rejects with `BadCertificateUriInvalid` on mismatch) | `urn:sientia:opc-ingestor` | +| `CN` | Certificate common name | `sientia-opc-ingestor` | +| `DAYS` | Certificate validity (days) | `3650` | +| `FORCE` | Set to `1` to overwrite an existing cert already pushed to the repo | unset | + +What the script does: +1. Generates a self-signed RSA-2048 X.509 cert + key with `openssl`, with the SAN URI, `keyUsage`, and `extendedKeyUsage` extensions OPC UA / KEPServer require. +2. Clones the ingestor's Gitea repo, copies the cert/key (`.pem`) and a `.der` copy (for manual import) into `certs/`, commits, and pushes. +3. Prints the paths the pod will see after cloning (`/app/certs/opc-ingestor-cert.pem`, `/app/certs/opc-ingestor-key.pem`) and the next manual steps. + +After pushing, you still need to: +- Set `cert_path` / `private_key_path` (and `uri`) on the server's document in the `OPC_servers` collection so `OpcManager` picks them up (see [OPC Server Configuration](#opc-server-configuration)). +- Restart the ingestor deployment so it re-clones the repo: `kubectl -n sientia rollout restart deployment/opc-ingestor`. +- On first connection, the server rejects the new cert — trust it in the OPC UA server's console (e.g. KEPServer: *OPC UA Configuration Manager → Trusted Clients → Trust*) and restart its runtime. + ## 🧪 Testing ### Unit Tests @@ -215,11 +248,9 @@ The OPC Ingestor exposes comprehensive Prometheus metrics for monitoring: - `ingestor_slots_managed_current`: Slots managed by this instance - `ingestor_active_total`: Total active ingestor instances - `redis_connection_status`: Redis connection health -- `kafka_connection_status`: Kafka connection health ### Data Processing Metrics - `ingestor_tag_written_count`: Data write operations -- `kafka_messages_sent_total`: Kafka message count - `redis_operations_total`: Redis operation count ## ⚙️ Configuration @@ -228,8 +259,6 @@ The OPC Ingestor exposes comprehensive Prometheus metrics for monitoring: | Variable | Description | Default | Required | |----------|-------------|---------|----------| -| `KAFKA_SERVERS` | Comma-separated Kafka server addresses | `localhost:9092` | No | -| `EXPORT_TO_KAFKA` | Enable Kafka data export | `false` | No | | `REDIS_HOST` | Redis server hostname | `localhost` | Yes | | `REDIS_PORT` | Redis server port | `6379` | Yes | | `REDIS_USERNAME` | Redis username | `None` | No | @@ -246,7 +275,7 @@ The OPC Ingestor exposes comprehensive Prometheus metrics for monitoring: ### OPC Server Configuration -OPC servers are configured through Redis with the following structure: +OPC servers are configured through Redis with the following structure. `cert_path` / `private_key_path` are only required for secured connections (see [Certificate Generation](#-certificate-generation)): ```json { @@ -256,6 +285,8 @@ OPC servers are configured through Redis with the following structure: "server_id": "1", "url": "opc.tcp://localhost:4841", "server_uri": "http://opcua-server.simulator", + "cert_path": "/app/certs/opc-ingestor-cert.pem", + "private_key_path": "/app/certs/opc-ingestor-key.pem", "tags": { "ns=2;i=2": { "aggr_func": "avg", @@ -292,6 +323,7 @@ sientia-dataops-opc-ingestor/ │ ├── ingestor.py # Core ingestor logic │ └── metrics.py # Prometheus metrics definitions ├── simulator/ # OPC simulation and testing tools +├── scripts/ # Ops scripts (e.g. OPC UA cert generation) ├── tests/ # Test suite ├── docker-compose.yaml # Infrastructure services └── requirements.txt # Python dependencies @@ -319,12 +351,7 @@ sientia-dataops-opc-ingestor/ - Check authentication credentials - Ensure proper network configuration -3. **Kafka Export Failures** - - Verify Kafka cluster is running - - Check broker addresses and network connectivity - - Review topic configuration and permissions - -4. **Performance Issues** +3. **Performance Issues** - Monitor Prometheus metrics for bottlenecks - Adjust polling intervals and lease TTLs - Review OPC server performance and network latency @@ -350,7 +377,7 @@ export LOG_LEVEL=DEBUG - **Horizontal Scaling**: Deploy multiple ingestor instances for high availability - **Load Distribution**: Use Redis-based slot allocation for automatic load balancing - **Resource Limits**: Monitor CPU, memory, and network usage -- **Database Performance**: Optimize MongoDB indexes and Kafka partitioning +- **Database Performance**: Optimize MongoDB indexes ## 🤝 Contributing