Merge branch 'feature/SIENTIAPDE-1972-gateway-write-read' into feature/SIENTIAPDE-1972-gateway-wiring

# Conflicts:
#	README.md
#	requirements.txt
This commit is contained in:
PedroHMCosme
2026-09-22 10:55:23 -03:00
18 changed files with 291 additions and 215 deletions

View File

@@ -46,7 +46,7 @@ The OPC Ingestor uses a modular, manager-based architecture designed for scalabi
- **Ingestor**: Main application orchestrator managing the overall lifecycle
- **IngestorManager**: Coordinates slot allocation, OPC server management, and load balancing
- **OPC Manager**: Handles individual OPC UA server connections and tag subscriptions
- **Data Manager**: Manages data streaming (MongoDB)
- **Data Manager**: Manages data persistence (MongoDB)
- **Resource Manager**: Coordinates resource allocation and instance coordination via Redis
## 🚪 OPC Gateway (SIENTIAPDE-1972)
@@ -123,6 +123,10 @@ health probe, NetworkPolicy, cert cache) and the `# OPC Gateway` block in
# Ensure services are accessible on localhost with appropriate ports
```
6. **Generate an OPC UA client certificate** (only if connecting to a secured OPC UA server)
See [Certificate Generation](#-certificate-generation) below.
## Usage
### Running the Ingestor
@@ -173,7 +177,7 @@ pytest
# Run with coverage
pytest --cov=ingestor --cov-report=html
```
### Populating Redis with OPC Configuration
@@ -189,6 +193,38 @@ pytest --cov=ingestor --cov-report=html
The feeder creates sample OPC tag configurations in Redis that the ingestor can discover and manage.
## 🔐 Certificate Generation
OPC UA servers that require secure connections (e.g. KEPServer with `Basic256` policy) validate the ingestor via an X.509 client certificate. Use `scripts/generate-and-push-opc-cert.sh` to generate that certificate and push it to the Gitea repo the pod clones on boot.
```bash
GITEA_PASSWORD='<senha-do-gitea>' ./scripts/generate-and-push-opc-cert.sh
```
Environment variables (all optional except `GITEA_PASSWORD`):
| Variable | Description | Default |
|----------|-------------|---------|
| `GITEA_URL` | Gitea in-cluster URL | `https://git.sientia.ai` |
| `GITEA_USER` | Gitea user | `gitea_admin` |
| `GITEA_PASSWORD` | Gitea password (**required**) | — |
| `GITEA_REPO` | Repo the ingestor pod clones | `gitea_admin/sientia-dataops-opc-ingestor` |
| `BRANCH` | Branch to push the cert to | `main` |
| `APP_URI` | SAN URI of the cert — **must match** the `uri` field of the corresponding document in the `OPC_servers` Mongo collection, since the ingestor uses it as `application_uri` (KEPServer rejects with `BadCertificateUriInvalid` on mismatch) | `urn:sientia:opc-ingestor` |
| `CN` | Certificate common name | `sientia-opc-ingestor` |
| `DAYS` | Certificate validity (days) | `3650` |
| `FORCE` | Set to `1` to overwrite an existing cert already pushed to the repo | unset |
What the script does:
1. Generates a self-signed RSA-2048 X.509 cert + key with `openssl`, with the SAN URI, `keyUsage`, and `extendedKeyUsage` extensions OPC UA / KEPServer require.
2. Clones the ingestor's Gitea repo, copies the cert/key (`.pem`) and a `.der` copy (for manual import) into `certs/`, commits, and pushes.
3. Prints the paths the pod will see after cloning (`/app/certs/opc-ingestor-cert.pem`, `/app/certs/opc-ingestor-key.pem`) and the next manual steps.
After pushing, you still need to:
- Set `cert_path` / `private_key_path` (and `uri`) on the server's document in the `OPC_servers` collection so `OpcManager` picks them up (see [OPC Server Configuration](#opc-server-configuration)).
- Restart the ingestor deployment so it re-clones the repo: `kubectl -n sientia rollout restart deployment/opc-ingestor`.
- On first connection, the server rejects the new cert — trust it in the OPC UA server's console (e.g. KEPServer: *OPC UA Configuration Manager → Trusted Clients → Trust*) and restart its runtime.
## 🧪 Testing
### Unit Tests
@@ -270,7 +306,7 @@ gateway-specific environment variables.
### OPC Server Configuration
OPC servers are configured through Redis with the following structure:
OPC servers are configured through Redis with the following structure. `cert_path` / `private_key_path` are only required for secured connections (see [Certificate Generation](#-certificate-generation)):
```json
{
@@ -280,6 +316,8 @@ OPC servers are configured through Redis with the following structure:
"server_id": "1",
"url": "opc.tcp://localhost:4841",
"server_uri": "http://opcua-server.simulator",
"cert_path": "/app/certs/opc-ingestor-cert.pem",
"private_key_path": "/app/certs/opc-ingestor-key.pem",
"tags": {
"ns=2;i=2": {
"aggr_func": "avg",
@@ -316,8 +354,8 @@ sientia-dataops-opc-ingestor/
│ ├── ingestor.py # Core ingestor logic
│ └── metrics.py # Prometheus metrics definitions
├── simulator/ # OPC simulation and testing tools
├── scripts/ # Ops scripts (e.g. OPC UA cert generation)
├── tests/ # Test suite
├── docker-compose.yaml # Infrastructure services
└── requirements.txt # Python dependencies
```
@@ -387,10 +425,6 @@ export LOG_LEVEL=DEBUG
- Use type hints where appropriate
- Follow the established architectural patterns
## 📄 License
This project is licensed under the terms specified in the LICENSE file.
## 🆘 Support
For support and questions:
@@ -402,14 +436,3 @@ For support and questions:
---
**Note**: This OPC Ingestor is designed for production use in industrial environments. Ensure proper security configuration and network isolation for production deployments.
This comprehensive documentation provides:
1. **Complete feature overview** with architectural details
2. **Detailed installation and setup instructions**
3. **Comprehensive configuration documentation**
4. **Performance tuning and troubleshooting guides**
5. **Development guidelines and contribution standards**
6. **Updated docstrings** for all major classes and methods
The documentation now serves as a complete reference for users, developers, and operators of the OPC Ingestor system.