102 online 3 members  + Join RP Base  +
FreeScript

runnerq-rust

by runnerq  ·  version 2026.10.02  ·  updated 3 days ago

No reviews yet

Downloads
0
Reviews
0
Size
886 KB
Code check
Clean
Licence
MIT
Released
Nov 2026

Durable Rust functions, with pluggable storage

runnerq-rust
Runner-Q
Durable Rust functions, with pluggable storage.
Features
  • Pluggable backend system - Trait-based storage abstraction; PostgreSQL is built-in; Redis available via the runner_q_redis crate
  • Priority-based activity processing - Support for Critical, High, Normal, and Low priority levels
  • Activity scheduling - Precise timestamp-based scheduling for future execution
  • Intelligent retry mechanism - Built-in retry mechanism with exponential backoff
  • Dead letter queue - Failed activities are moved to a dead letter queue for inspection
  • Concurrent activity processing - Configurable number of concurrent workers
  • Graceful shutdown - Proper shutdown handling with signal support
  • Activity orchestration - Activities can execute other activities for complex workflows
  • Comprehensive error handling - Retryable and non-retryable error types
  • Activity metadata - Support for custom metadata on activities
  • Built-in observability console - Real-time web UI for monitoring and managing activities
  • Worker-level activity type filtering - Isolate workloads by restricting each engine to specific activity types
  • Queue statistics - Monitoring capabilities and metrics collection
Storage Backends
Backend, Status, Use Case

PostgreSQL, ✅ Built-in, Default. Permanent persistence, SQL-based queries
Redis, ✅ Optional, Use the runner_q_redis crate for Redis or Valkey
Custom, ✅ Supported, Implement Storage trait for your own backend
Installation
cargo add runner_q

Quick Start
use runner_q::{WorkerEngine, ActivityPriority, ActivityHandler, ActivityContext, ActivityHandlerResult, ActivityError};
use std::sync::Arc;
use async_trait::async_trait;
use serde_json::json;
use serde::{Serialize, Deserialize};
use std::time::Duration;

// Define activity types
**[derive(Debug, Clone)]**
enum MyActivityType {
    SendEmail,
    ProcessPayment,
}

impl std::fmt::Display for MyActivityType {
    fn fmt(&self, f: &mut std::fmt::Formatter|".backend()"| QB
    WEB -->|".redis_url()"| RB
    WE --> QB
    WE --> IB
    RB --> QB
    RB --> IB
    PB --> QB
    PB --> IB
    Future -.-> QB
    Future -.-> IB

The public API remains unchanged - you can continue using .redis_url() for the default experience, or use .backend() to inject a custom implementation.
Using the Default Redis Backend
use runner_q::{WorkerEngine, RedisBackend};
use std::sync::Arc;

// Option 1: Use the simple redis_url API (recommended for most cases)
let engine = WorkerEngine::builder()
    .redis_url("redis://localhost:6379")
    .queue_name("my_app")
    .build()
    .await?;

// Option 2: Create a RedisBackend explicitly for more control
let backend = RedisBackend::builder()
    .redis_url("redis://localhost:6379")
    .queue_name("my_app")
    .lease_ms(60_000)  // Custom lease duration
    .build()
    .await?;

let engine = WorkerEngine::builder()
    .backend(Arc::new(backend))
    .max_workers(8)
    .build()
    .await?;

Valkey Compatibility
Since Valkey is Redis protocol-compatible, you can use it directly by pointing the URL to your Valkey server:

let engine = WorkerEngine::builder()
    .redis_url("redis://valkey-server:6379")  // Works with Valkey!
    .queue_name("my_app")
    .build()
    .await?;

PostgreSQL Backend
For use cases requiring permanent persistence and SQL-based queries, RunnerQ provides a PostgreSQL backend:

**Cargo.toml**
[dependencies]
runner_q = { version = "0.5", features = ["postgres"] }

use runner_q::storage::PostgresBackend;
use std::sync::Arc;

// Create PostgreSQL backend
let backend = Arc::new(
    PostgresBackend::new(
        "postgres://user:password@localhost/runnerq",
        "my_queue"
    ).await?
);

// Use with WorkerEngine
let engine = WorkerEngine::builder()
    .backend(backend)
    .max_workers(8)
    .build()
    .await?;

PostgreSQL Backend Features:
  • Permanent Persistence - Activities stored indefinitely (no TTL expiration)
  • Multi-node Safe - Uses FOR UPDATE SKIP LOCKED for concurrent job claiming
  • Cross-process Events - PostgreSQL LISTEN/NOTIFY for real-time event streaming
  • Atomic Idempotency - Separate table with INSERT ... ON CONFLICT for race-safe key claiming
  • History Preservation - Never deletes activity records

Schema Tables Created:
  • runnerq_activities - Main activity storage
  • runnerq_events - Event history timeline
  • runnerq_results - Activity execution results
  • runnerq_idempotency - Idempotency key mapping

See examples/postgres_example.rs for a complete working example, and examples/activity_type_filtering.rs for workload isolation with multiple engines.
Implementing a Custom Backend
You can implement your own backend by implementing the Storage trait (which combines QueueStorage and InspectionStorage):

```rust
use runner_q::storage::{
Storage, QueueStorage, InspectionStorage, StorageError,
QueuedActivity, DequeuedActivity, FailureKind,
};
use runner_q::{QueueStats, ActivitySnapshot, ActivityEvent, DeadLetterRecord};
use async_trait::async_trait;
use std::time::Duration;
use uuid::Uuid;

pub struct MyCustomBackend {
// Your backend state (connection pool, config, etc.)
}
[async_trait]
impl QueueStorage for MyCustomBackend {
async fn enqueue(&self, activity: QueuedActivity) -> Result {
// Implement activity enqueuing
todo!()
}

async fn dequeue(
&self,
worker_id: &str,
timeout: Duration,
activity_types: Option,
) -> Result {
// Implement activity claiming.
// When activity_types is Some, only claim matching types.
todo!()
}

async fn ack_success(
&self,
activity_id: Uuid,
lease_id: &str,
result: Option,
worker_id: Option,
) -> Result {
// Mark activity as completed
todo!()

…