Protocol-Implementation-Roadmap
Protocol Implementation Roadmap
Date: 2025-11-12
Status: 📋 Planning Document
Purpose: Step-by-step guide for implementing WhatNext P2P protocols
Overview
This document provides a detailed roadmap for implementing custom libp2p protocols in WhatNext. Each protocol builds on the previous one, teaching progressively more complex patterns.
Protocol 1: Handshake (/whatnext/handshake/1.0.0)
Purpose
Establish peer identity and capabilities when connection is made.
What You'll Learn
- How to register a protocol handler in libp2p
- Stream-based communication (read/write)
- Message framing (length-prefix or newline-delimited)
- Request/response pattern
- Error handling in streams
Implementation Steps
-
Define Protocol in Config
// app/src/shared/p2p-config.ts PROTOCOLS: { HANDSHAKE: '/whatnext/handshake/1.0.0', } -
Create Protocol Handler Module
// app/src/utility/protocols/handshake.ts export async function handshakeProtocolHandler(stream) { // Read incoming handshake // Send our handshake // Return peer metadata } -
Register Handler in P2P Service
// app/src/utility/p2p-service.ts (in startNode) await this.libp2pNode.handle( P2P_CONFIG.PROTOCOLS.HANDSHAKE, handshakeProtocolHandler ); -
Initiate Handshake on Connection
// In peer:connect event handler const stream = await this.libp2pNode.dialProtocol( peerId, P2P_CONFIG.PROTOCOLS.HANDSHAKE ); // Send our handshake, receive theirs -
Update UI to Show Metadata
- Display peer's app version
- Display peer's display name (from handshake, not auto-generated)
- Display peer's capabilities
Success Criteria
- Handshake sent immediately after connection
- Peer display name shows real name (not "Peer 12D3…")
- App version visible in peer details
- Capabilities list populated
- Error handling for handshake timeout/failure
Example Message Format
{
"type": "handshake",
"payload": {
"displayName": "Mads' WhatNext",
"appVersion": "0.0.0",
"protocolVersion": "1.0.0",
"capabilities": ["playlist-sync", "file-transfer"]
},
"timestamp": "2025-11-12T10:30:00Z"
}
Helpful Resources
- libp2p Stream Muxing
- Protocol Handlers
- it-length-prefixed - Message framing library
Protocol 2: Data Test (/whatnext/data-test/1.0.0)
Purpose
Send arbitrary data between peers to understand streaming and chunking.
What You'll Learn
- Sending larger payloads (1KB - 10MB)
- Stream backpressure handling
- Progress callbacks
- Chunking strategies
- Performance measurement
Implementation Steps
-
Add Data Test UI Controls
// In P2PStatus.tsx "Data Transfer Testing" section <button onClick={() => sendTestMessage(peer, size)}> Send {size} Test </button> -
Create Protocol Handler
// app/src/utility/protocols/data-test.ts export async function dataTestHandler(stream) { // Read test data // Optionally echo back if requested // Track bytes received } -
Implement Send Logic
async function sendTestData(peerId, dataSize, echo = false) { const stream = await libp2p.dialProtocol( peerId, '/whatnext/data-test/1.0.0' ); // Generate test data // Send in chunks // If echo, wait for response // Measure time/throughput } -
Add Statistics Tracking
- Bytes sent/received per peer
- Transfer time
- Throughput (bytes/sec)
- Display in UI
Test Scenarios
- Send 1KB message (small, instant)
- Send 1MB message (medium, test chunking)
- Send 10MB message (large, test progress)
- Echo test (send, receive same data back)
- Concurrent sends (test backpressure)
Success Criteria
- Small messages send instantly
- Large messages show progress
- Statistics accurate (match actual data size)
- Echo returns identical data
- No memory leaks on large transfers
Example UI Addition
const testSizes = [
{ label: '1 KB', bytes: 1024 },
{ label: '100 KB', bytes: 102400 },
{ label: '1 MB', bytes: 1048576 },
{ label: '10 MB', bytes: 10485760 },
];
Protocol 3: File Transfer (/whatnext/file-transfer/1.0.0)
Purpose
Transfer actual files between peers with progress and error recovery.
What You'll Learn
- File reading/writing
- Chunked transfer with resume support
- Progress callbacks
- Error recovery patterns
- File metadata exchange
Implementation Steps
-
Add File Transfer UI
<button onClick={selectFileToSend}> Select File to Send </button> -
File Metadata Exchange
// First message: metadata { "fileName": "album-art.jpg", "fileSize": 1048576, "mimeType": "image/jpeg", "checksum": "sha256:abc123..." } // Then: chunks { "chunkIndex": 0, "totalChunks": 100, "data": "base64encodeddata..." } -
Implement Chunking Logic
const CHUNK_SIZE = 64 * 1024; // 64KB chunks async function sendFile(peerId, filePath) { // Read file metadata // Send metadata first // Read file in chunks // Send each chunk with index // Wait for ack/completion } -
Progress Tracking
- Show file name, size
- Progress bar (percent complete)
- Transfer rate (MB/s)
- ETA
- Cancel button
Test Scenarios
- Send small file (< 1MB)
- Send medium file (1-10 MB)
- Send large file (> 10 MB)
- Verify file integrity (checksum)
- Cancel mid-transfer
- Multiple concurrent transfers
Success Criteria
- File arrives intact (checksums match)
- Progress updates smoothly
- Can handle large files (100MB+)
- Cancel works cleanly
- No corrupted transfers
Error Handling
- Connection drops mid-transfer (resume or restart?)
- Disk full on receiver
- File permissions issues
- Invalid file path
Protocol 4: Playlist Sync (/whatnext/playlist-sync/1.0.0)
Purpose
Synchronize playlist changes between peers in real-time.
What You'll Learn
- CRDT concepts (Conflict-Free Replicated Data Types)
- RxDB replication protocol
- Conflict resolution strategies
- Event-driven updates
- Operational transformation
Implementation Steps
-
Integrate RxDB Replication
// Use RxDB's built-in P2P replication import { replicateRxCollection } from 'rxdb/plugins/replication'; -
Create Sync Protocol Handler
// This might be simpler - RxDB handles most of it // You provide: // - Connection/stream management // - Peer discovery // RxDB handles: // - Diff/patch generation // - Conflict resolution // - Eventual consistency -
Test Scenarios
- Create playlist on peer A, appears on peer B
- Add track on peer A, appears on peer B
- Concurrent adds (both peers add at same time)
- Concurrent deletes (conflict resolution)
- Rename playlist while other peer adds track
-
Observe CRDT Behavior
- Add debug logging for sync events
- Show "last synced" timestamp per peer
- Display conflict resolution decisions
Success Criteria
- Changes replicate within 1-2 seconds
- No data loss during conflicts
- Order preserved for sequential ops
- Convergence (all peers eventually agree)
This is Complex
Playlist sync is the most sophisticated protocol. It builds on all previous learnings:
- Handshake → establishes peer identity
- Data test → proves large data works
- File transfer → proves chunking/progress works
- RxDB replication → puts it all together
Don't rush to this one. Master the simpler protocols first.
General Implementation Pattern
Every protocol follows this structure:
1. Protocol Registration
// In p2p-service.ts startNode()
await this.libp2pNode.handle(
'/whatnext/my-protocol/1.0.0',
myProtocolHandler
);
2. Protocol Handler
async function myProtocolHandler({ stream, connection }) {
try {
// Read incoming data
const data = await readFromStream(stream);
// Process
const response = processData(data);
// Write response (optional)
await writeToStream(stream, response);
// Close stream
await stream.close();
} catch (error) {
console.error('Protocol error:', error);
stream.abort(error);
}
}
3. Protocol Dialing
async function initiateProtocol(peerId, data) {
const stream = await this.libp2pNode.dialProtocol(
peerId,
'/whatnext/my-protocol/1.0.0'
);
// Write request
await writeToStream(stream, data);
// Read response
const response = await readFromStream(stream);
// Close
await stream.close();
return response;
}
4. Stream Helpers
You'll want to create utilities for:
- Reading length-prefixed messages
- Writing length-prefixed messages
- JSON serialization/deserialization
- Error handling/timeouts
Example:
// app/src/utility/protocols/stream-utils.ts
export async function readJSON(stream) {
const pipe = stream.source
| lpDecode() // length-prefix decode
| toBuffer()
| toString();
for await (const data of pipe) {
return JSON.parse(data);
}
}
export async function writeJSON(stream, obj) {
const data = JSON.stringify(obj);
await stream.sink([
lpEncode()(Buffer.from(data))
]);
}
Tips for Success
Start Simple
- Don't try to build all protocols at once
- Each one teaches something new
- Test thoroughly before moving on
Use the Dev UI
- Add protocol-specific sections as you go
- Log every message sent/received
- Display statistics
Handle Errors
- Streams can fail
- Connections can drop
- Timeouts are important
Document Learnings
- Keep writing notes
- Capture "gotchas"
- Share insights
Test with Multiple Peers
- Use test-peer CLI
- Start multiple WhatNext instances
- Test concurrent operations
Estimated Timeline
Realistic pace (assuming 2-4 hours per protocol):
- Week 1: Handshake protocol (simplest, learn foundations)
- Week 2: Data test protocol (build on handshake, learn performance)
- Week 3: File transfer protocol (chunking, progress, error handling)
- Week 4-5: Playlist sync (most complex, integrates everything)
Aggressive pace (full-time work):
- Could complete all in 1-2 weeks
No rush! The goal is understanding, not speed.
Success Metrics
By the end of this roadmap, you should be able to:
- Explain how libp2p protocol handlers work
- Implement a custom protocol from scratch
- Debug stream-based communication issues
- Measure and optimize data transfer performance
- Integrate RxDB replication
- Confidently build new protocols for future features
Next: Handshake Protocol
Start with Protocol 1: Handshake above. It's the simplest and most important. Everything else builds on it.
Good luck! 🚀