Data Flow Architecture

This document describes how data flows through WsprDaemon from audio capture to final spot reporting.

Overview

WsprDaemon processes WSPR signals through a multi-stage pipeline that transforms raw audio into validated spot reports. The architecture is designed for reliability, scalability, and data integrity.

[SDR Hardware] → [Audio Capture] → [Decoding] → [Validation] → [Merging] → [Upload]

Stage 1: Audio Capture

Input Sources

  • KiwiSDR: Network-based audio streams via kiwirecorder.py

  • RX888: USB-based I/Q data via ka9q-radio

  • RTL-SDR: USB-based I/Q data via rtl_sdr

  • Other SDRs: Various interfaces through standardized APIs

Recording Process

  1. Stream Initialization: Establish connection to SDR hardware

  2. Audio Buffering: Continuous audio capture with 2-minute windows

  3. File Creation: Generate timestamped WAV files (YYMMDD_HHMM.wav)

  4. Quality Control: Monitor for clipping, dropouts, and timing accuracy

Data Format

  • Sample Rate: 12 kHz (WSPR standard)

  • Bit Depth: 16-bit signed integer

  • Channels: Mono (single channel)

  • Duration: Exactly 120 seconds per file

  • Timing: Synchronized to even-minute boundaries

Stage 2: Signal Decoding

Decoding Engine

  • Primary Decoder: wsprd from WSJT-X suite

  • Deep Search: Optional wsprd -d for enhanced sensitivity

  • Parallel Processing: Multiple decoders per receiver/band combination

Processing Pipeline

  1. File Detection: Monitor for new WAV files

  2. Preprocessing: Audio validation and conditioning

  3. WSPR Decoding: Extract callsign, grid, power, frequency, SNR

  4. Noise Analysis: Calculate background noise levels

  5. Quality Assessment: Validate decoded parameters

Output Data

Timestamp: 2024-01-15 12:34:00
Callsign: W1ABC
Grid: FN42
Power: 37 dBm
Frequency: 14.097123 MHz
SNR: -18 dB
Drift: 0 Hz/min

Stage 3: Data Validation

Validation Layers

  1. Format Validation: Verify callsign, grid, power formats

  2. Range Checking: Ensure values within expected bounds

  3. Temporal Validation: Confirm timing alignment

  4. Cross-Reference: Compare with previous observations

Quality Metrics

  • Decode Confidence: Statistical confidence in decoded parameters

  • Signal Quality: SNR and frequency stability metrics

  • Timing Accuracy: Synchronization with WSPR schedule

  • Consistency: Agreement with historical patterns

Stage 4: Multi-Receiver Processing

Spot Merging

When multiple receivers monitor the same band:

  1. Collection: Gather spots from all receivers

  2. Correlation: Match spots by callsign, time, and frequency

  3. Selection: Choose best SNR for each unique transmission

  4. Metadata: Preserve information from all receivers

Receiver Coordination

  • Time Synchronization: Ensure all receivers use common time reference

  • Frequency Calibration: Account for receiver-specific offsets

  • Performance Monitoring: Track individual receiver health

  • Load Balancing: Distribute processing across available resources

Stage 5: Data Aggregation

Spot Compilation

  1. Temporal Grouping: Collect spots by 2-minute transmission windows

  2. Duplicate Removal: Eliminate redundant observations

  3. Metadata Addition: Add receiver, location, and system information

  4. Format Conversion: Prepare data for various output formats

Noise Data Processing

  • Calibration: Apply receiver-specific corrections

  • Averaging: Calculate time-averaged noise levels

  • Trend Analysis: Identify patterns and anomalies

  • Graphing: Generate visualization data

Stage 6: External Reporting

Upload Destinations

  1. wsprnet.org: Primary WSPR spot database

  2. wsprdaemon.org: Enhanced data with additional metrics

  3. pskreporter.info: FT4/FT8 mode reporting

  4. HamSCI GRAPE: Scientific ionospheric research data

Upload Process

  1. Queue Management: Maintain upload queues per destination

  2. Format Conversion: Transform data to required formats

  3. Transmission: HTTP POST to destination servers

  4. Confirmation: Verify successful delivery

  5. Retry Logic: Handle temporary failures with exponential backoff

Data Formats

wsprnet.org Format:

240115 1234 W1ABC FN42 37 K1DEF FN32 -18 0 0 14.097123 1

Enhanced Format (wsprdaemon.org):

{
  "timestamp": "2024-01-15T12:34:00Z",
  "tx_call": "W1ABC",
  "tx_grid": "FN42",
  "tx_power": 37,
  "rx_call": "K1DEF",
  "rx_grid": "FN32",
  "snr": -18,
  "frequency": 14.097123,
  "drift": 0,
  "noise_level": -142.5,
  "receiver_id": "kiwi_01"
}

Data Storage and Archival

Local Storage

  • Active Processing: /tmp/wsprdaemon/ (tmpfs recommended)

  • Persistent Logs: ~/wsprdaemon/logs/

  • Archive Data: ~/wsprdaemon/archives/

  • Configuration: ~/wsprdaemon/wsprdaemon.conf

Data Retention

  • WAV Files: Deleted after successful processing (configurable)

  • Spot Data: Retained locally for backup and analysis

  • Log Files: Rotated based on size and age limits

  • Noise Data: Archived for long-term trend analysis

Error Handling and Recovery

Fault Tolerance

  1. Process Monitoring: Watchdog processes restart failed components

  2. Data Caching: Local storage until successful upload

  3. Graceful Degradation: Continue operation with reduced functionality

  4. Automatic Recovery: Self-healing from transient failures

Error Propagation

  • Logging: Comprehensive error logging at each stage

  • Alerting: Notification of critical failures

  • Metrics: Performance and error rate monitoring

  • Diagnostics: Built-in tools for troubleshooting

Performance Characteristics

Throughput

  • Single Band: ~30 spots/hour typical

  • Multi-Band: Scales linearly with receiver count

  • Peak Load: 2-minute processing windows create periodic load spikes

  • Sustained Rate: Designed for 24/7 continuous operation

Latency

  • Processing Delay: 2-4 minutes from transmission to upload

  • Upload Latency: Typically <30 seconds to external services

  • End-to-End: 3-5 minutes from transmission to public availability

Resource Usage

  • CPU: Moderate during decode windows, low otherwise

  • Memory: 200-500MB per active band

  • Disk I/O: Burst writes during WAV creation and processing

  • Network: Steady upload traffic, multicast for ka9q-radio

Scalability Considerations

Horizontal Scaling

  • Multiple Receivers: Linear scaling with receiver count

  • Distributed Processing: Separate recording and processing systems

  • Load Distribution: Balance processing across available cores

  • Network Optimization: Multicast for efficient data distribution

Vertical Scaling

  • CPU Optimization: Multi-threaded processing where beneficial

  • Memory Management: Efficient buffer management and cleanup

  • I/O Optimization: tmpfs for high-frequency file operations

  • Network Tuning: Optimized for upload patterns and retry logic

This architecture ensures reliable, scalable processing of WSPR data while maintaining the flexibility to adapt to different hardware configurations and operational requirements.