org-social/relay
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Compare commits

base: org-social:v1.0
Branches Tags
org-social:main
org-social:experiment/sse-notifications
org-social:develop
org-social:add-rss
org-social:v1.6
org-social:v1.2
org-social:v1.1
org-social:v1.0
...
compare: org-social:main
Branches Tags
org-social:main
org-social:experiment/sse-notifications
org-social:develop
org-social:add-rss
org-social:v1.6
org-social:v1.2
org-social:v1.1
org-social:v1.0

No commits in common. "v1.0" and "main" have entirely different histories.
v1.0 ... main

113 changed files with 10572 additions and 367 deletions
Show stats Expand all files Collapse all files

7
CONTRIBUTING.md Normal file
View file

@ -0,0 +1,7 @@
# Contributing
Thank you for your interest in contributing to this project!
For detailed contribution guidelines, please visit:
https://git.andros.dev/andros/contribute

2
Dockerfile
View file

@ -27,4 +27,4 @@ COPY . .
EXPOSE 8000
# Default command (can be overridden in docker-compose)
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
CMD ["uvicorn", "core.asgi:application", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]

666
README.md
View file

@ -2,14 +2,13 @@
## Introduction
Org Social Relay is a P2P system that acts as an intermediary between all [Org Social](https://github.com/tanrax/org-social) files. It scans the network, creating an index of users, mentions, replies, groups and threads. This allows you to:
Org Social Relay is a P2P system that acts as an intermediary between all [Org Social](https://git.andros.dev/org-social/org-social) files. It scans the network, creating an index of users, mentions, replies, groups and threads. This allows you to:
```mermaid
graph TD
List["📋 List nodes"]
Node1["🖥️ Node 1"]
Node2["🖥️ Node 2"]
Node3["🖥️ Node 3"]
Node1["🖥️ Relay"]
Node2["🖥️ Relay"]
Node3["🖥️ Relay"]
%% Social.org instances with icons
Social1_1["📄 social.org"]
@ -19,11 +18,6 @@ graph TD
Social3_1["📄 social.org"]
Social3_2["📄 social.org"]
%% Parent-child connections with labels
List -.->|"Get"| Node1
List -.->|"Get"| Node2
List -.->|"Get"| Node3
%% Node to social.org connections
Social1_1 -->|"⚓ Connects"| Node1
Social1_2 -->|"⚓ Connects"| Node1
@ -33,19 +27,17 @@ graph TD
Social3_2 -->|"⚓ Connects"| Node3
%% Bidirectional connections between nodes
Node1 <-.->|"👥 Share Users"| Node2
Node2 <-.->|"👥 Share Users"| Node3
Node1 <-.->|"👥 Share Users"| Node3
Node1 <-.->|"👥 Share Feeds"| Node2
Node2 <-.->|"👥 Share Feeds"| Node3
Node1 <-.->|"👥 Share Feeds"| Node3
%% Modern color scheme with gradients
classDef socialStyle fill:#667eea,stroke:#764ba2,stroke-width:3px,color:#fff,font-weight:bold
classDef nodeStyle fill:#f093fb,stroke:#f5576c,stroke-width:3px,color:#fff,font-weight:bold
classDef listStyle fill:#4facfe,stroke:#00f2fe,stroke-width:4px,color:#fff,font-weight:bold
%% Apply styles
class Social1_1,Social1_2,Social2_1,Social2_2,Social3_1,Social3_2 socialStyle
class Node1,Node2,Node3 nodeStyle
class List listStyle
```
[Source](/diagram.mmd)
@ -54,12 +46,14 @@ graph TD
- Read or participate in threads.
- Perform searches (tags and full text).
- Participate in groups.
- See who boosted your posts.
- Notify external sites linked from posts via [Webmention](https://www.w3.org/TR/webmention/) (sending only; receiving them is not possible since plain text feeds cannot advertise an endpoint).
- Follow Mastodon/ActivityPub accounts and RSS/Atom feeds from any Org Social client through [bridges](#bridges-follow-activitypub-accounts-and-rss-feeds) (virtual social.org feeds).
## Concepts
- **List nodes**: Index of public nodes. Simple list with all the URLs of the Nodes (`https://cdn.jsdelivr.net/gh/tanrax/org-social/org-social-relay-list.txt`). It will be used by nodes to find other nodes and share information.
- **Node**: A server running Org Social Relay (this software). It scans the network and shares information with other nodes or clients.
- **Client**: An application that connects to a Node to get information. It can be Org Social or any other application that implements the Org Social Relay API.
- **Relay**: A server running Org Social Relay (this software). Each relay maintains a local `relay-list.txt` file with URLs of other relays. Relays scan the network and share feed information with each other, creating a decentralized P2P network.
- **Client**: An application that connects to a Relay to get information. It can be Org Social or any other application that implements the Org Social Relay API.
## Installation
@ -92,13 +86,61 @@ nano .env
docker compose up -d
```
## Make your Org Social Relay public
## Managing Relay Nodes
If you want your Relay to be used by other users, and also communicate with other public Relays to work together scanning the network and improving everyone's speed, you must make a Pull Request to this file:
### Local Relay List
https://github.com/tanrax/org-social/blob/main/org-social-relay-list.txt
This relay uses a local file (`relay-list.txt` in the project root) to manage the list of other relay nodes it connects to.
Add your Relay URL (e.g. `https://my-relay.example.com`) in a new line.
To add or remove relay nodes from your local instance:
```bash
# Edit the relay list file
nano relay-list.txt
# Add one relay URL per line, for example:
# https://relay.org-social.org
# https://other-relay.example.com
```
The relay will automatically discover feeds from these nodes every 3 hours.
### Make your Org Social Relay public
If you want your Relay to be discoverable by other relay nodes, you can make a Pull Request to add your relay URL to the `relay-list.txt` file in this repository:
**Steps to add your relay:**
1. Fork this repository
2. Edit `relay-list.txt` and add your relay URL on a new line:
```
https://your-relay.example.com
```
3. Create a Pull Request with your changes
4. Once merged, all relay nodes that update their code (via `git pull`) will automatically discover your relay
This way, the network of relays grows organically as new nodes are added to the shared list.
## Updating
To update your Org Social Relay to the latest version:
```bash
# Navigate to your installation directory
cd /path/to/org-social-relay
# Check if Docker containers are running
docker compose up -d --build
# Pull the latest changes
git pull
# Apply database migrations (if any)
docker compose exec django python manage.py migrate
# Restart the services
docker compose restart
```
## Endpoints for clients
@ -114,6 +156,36 @@ You can use:
- Manual encoding: `curl "http://localhost:8080/endpoint/?param=encoded_url"`
- curl's automatic encoding: `curl -G "http://localhost:8080/endpoint/" --data-urlencode "param=unencoded_url"`
### HTTP Caching
All Relay endpoints that return data include HTTP caching headers:
- **`ETag`**: A unique identifier for the current state of the relay (e.g., `"a1b2c3d4"`). This value changes when the relay scans feeds for updates.
- **`Last-Modified`**: The timestamp when the relay last scanned feeds (e.g., `Wed, 01 Nov 2025 10:15:00 GMT`).
All endpoints return the same `ETag` and `Last-Modified` values, which represent the global state of the relay. These headers are updated by the periodic feed scanning task.
**Example:**
```sh
curl -i http://localhost:8080/mentions/?feed=https://example.com/social.org
# Response headers include:
# ETag: "abc123"
# Last-Modified: Wed, 01 Nov 2025 10:15:00 GMT
```
### CORS (Cross-Origin Resource Sharing)
All Relay endpoints have CORS enabled with `Access-Control-Allow-Origin: *`, allowing direct access from any frontend/web application. This means you can call the API directly from JavaScript in the browser without CORS restrictions.
**Example:**
```javascript
// Fetch notifications directly from the browser
fetch('http://localhost:8080/notifications/?feed=https://example.com/social.org')
.then(response => response.json())
.then(data => console.log(data));
```
### Root
`/` - Basic information about the relay.
@ -134,16 +206,26 @@ curl http://localhost:8080/
"self": {"href": "/", "method": "GET"},
"feeds": {"href": "/feeds/", "method": "GET"},
"add-feed": {"href": "/feeds/", "method": "POST"},
"feed-content": {"href": "/feed-content/?feed={feed_url}", "method": "GET", "templated": true},
"notifications": {"href": "/notifications/?feed={feed_url}", "method": "GET", "templated": true},
"sse-notifications": {"href": "/sse/notifications/?feed={feed_url}", "method": "GET", "templated": true},
"sse-notifications-all": {"href": "/sse/notifications/", "method": "GET"},
"mentions": {"href": "/mentions/?feed={feed_url}", "method": "GET", "templated": true},
"reactions": {"href": "/reactions/?feed={feed_url}", "method": "GET", "templated": true},
"replies-to": {"href": "/replies-to/?feed={feed_url}", "method": "GET", "templated": true},
"boosts": {"href": "/boosts/?post={post_url}", "method": "GET", "templated": true},
"interactions": {"href": "/interactions/?post={post_url}", "method": "GET", "templated": true},
"replies": {"href": "/replies/?post={post_url}", "method": "GET", "templated": true},
"search": {"href": "/search/?q={query}", "method": "GET", "templated": true},
"groups": {"href": "/groups/", "method": "GET"},
"group-messages": {"href": "/groups/{group_slug}/", "method": "GET", "templated": true},
"polls": {"href": "/polls/", "method": "GET"},
"poll-votes": {"href": "/polls/votes/?post={post_url}", "method": "GET", "templated": true}
"poll-votes": {"href": "/polls/votes/?post={post_url}", "method": "GET", "templated": true},
"profile": {"href": "/profile/?feed={feed_url}", "method": "GET", "templated": true},
"rss": {"href": "/rss.xml", "method": "GET", "description": "RSS feed of latest posts (supports ?tag={tag} and ?feed={feed_url} filters)"},
"bridge": {"href": "/bridge/", "method": "GET"},
"bridge-activitypub": {"href": "/bridge/activitypub/@{user}@{instance}/", "method": "GET", "templated": true},
"bridge-rss": {"href": "/bridge/rss/?url={feed_url}", "method": "GET", "templated": true}
}
}
```
@ -191,7 +273,7 @@ curl -X POST http://localhost:8080/feeds/ -d '{"feed": "https://example.com/path
### Get notifications
`/notifications/?feed={url feed}` - Get all notifications (mentions, reactions, and replies) received by a given feed. Results are ordered from most recent to oldest.
`/notifications/?feed={url feed}` - Get all notifications (mentions, reactions, replies, and boosts) received by a given feed. Results are ordered from most recent to oldest.
```sh
# URL must be encoded when passed as query parameter
@ -206,6 +288,11 @@ curl -G "http://localhost:8080/notifications/" --data-urlencode "feed=https://ex
"type": "Success",
"errors": [],
"data": [
{
"type": "boost",
"post": "https://alice.org/social.org#2025-02-05T14:00:00+0100",
"boosted": "https://example.com/social.org#2025-02-05T10:00:00+0100"
},
{
"type": "reaction",
"post": "https://alice.org/social.org#2025-02-05T13:15:00+0100",
@ -230,13 +317,13 @@ curl -G "http://localhost:8080/notifications/" --data-urlencode "feed=https://ex
],
"meta": {
"feed": "https://example.com/social.org",
"total": 4,
"total": 5,
"by_type": {
"mentions": 1,
"reactions": 2,
"replies": 1
},
"version": "123"
"replies": 1,
"boosts": 1
}
},
"_links": {
"self": {"href": "/notifications/?feed=https%3A%2F%2Fexample.com%2Fsocial.org", "method": "GET"},
@ -248,24 +335,84 @@ curl -G "http://localhost:8080/notifications/" --data-urlencode "feed=https://ex
```
Each notification includes:
- `type`: The notification type (`"mention"`, `"reaction"`, or `"reply"`)
- `type`: The notification type (`"mention"`, `"reaction"`, `"reply"`, or `"boost"`)
- `post`: The notification post URL (format: `{author_feed}#{timestamp}`)
- `emoji`: (Only for reactions) The reaction emoji
- `parent`: (Only for reactions and replies) The post URL that received the notification
- `boosted`: (Only for boosts) The original post URL that was boosted
**Mentions** only have `type` and `post` because you are the one being mentioned in someone else's post.
**Reactions and replies** have `parent` to indicate which of your posts received the reaction/reply.
To extract the author's feed from the `post` field, simply take the part before the `#` character. For example, from `https://alice.org/social.org#2025-02-05T13:15:00+0100`, the author is `https://alice.org/social.org`.
**Boosts** have `boosted` to indicate which of your posts was shared.
The `version` in the `meta` field is a unique identifier for the current state of all notifications. You can use it to check if there are new notifications since your last request.
To extract the author's feed from the `post` field, simply take the part before the `#` character. For example, from `https://alice.org/social.org#2025-02-05T13:15:00+0100`, the author is `https://alice.org/social.org`.
The `by_type` breakdown in `meta` allows you to show notification counts per type in your UI.
**Optional parameters:**
- `type`: Filter by notification type (`mention`, `reaction`, `reply`)
- `type`: Filter by notification type (`mention`, `reaction`, `reply`, `boost`)
- Example: `/notifications/?feed={feed}&type=reaction`
- Example: `/notifications/?feed={feed}&type=boost`
### Real-time notifications (SSE)
`/sse/notifications/` - Subscribe to real-time notifications via Server-Sent Events. The `?feed=` parameter is optional:
- **With `?feed=`**: streams only notifications for that feed.
- **Without `?feed=`**: streams all notifications from all feeds (intended for backends such as push notification services).
```sh
# Per-feed stream
curl -N "http://localhost:8080/sse/notifications/?feed=https://example.com/social.org"
# Global stream (all feeds)
curl -N "http://localhost:8080/sse/notifications/"
```
**Events:**
- `connected` - Connection established
```
event: connected
data: {"feed": "https://example.com/social.org", "status": "connected"}
```
In the global stream the `feed` field is omitted:
```
event: connected
data: {"status": "connected"}
```
- `heartbeat` - Connection keepalive (every 30s)
```
event: heartbeat
data: {"status": "alive", "timestamp": 1733392800}
```
- `notification` - New notification received (per-feed, same structure as `/notifications/`)
```
event: notification
data: {"type": "mention", "post": "https://alice.org/social.org#2025-02-05T11:20:00+0100"}
```
```
event: notification
data: {"type": "reply", "post": "https://bob.org/social.org#...", "parent": "https://example.com/social.org#..."}
```
```
event: notification
data: {"type": "reaction", "post": "https://carol.org/social.org#...", "emoji": "❤", "parent": "https://example.com/social.org#..."}
```
```
event: notification
data: {"type": "boost", "post": "https://dave.org/social.org#...", "boosted": "https://example.com/social.org#..."}
```
In the global stream, each notification includes a `target_feed` field identifying which feed received it:
```
event: notification
data: {"target_feed": "https://example.com/social.org", "type": "mention", "post": "https://alice.org/social.org#2025-02-05T11:20:00+0100"}
```
### Get mentions
@ -290,8 +437,7 @@ curl -G "http://localhost:8080/mentions/" --data-urlencode "feed=https://example
],
"meta": {
"feed": "https://example.com/social.org",
"total": 3,
"version": "123"
"total": 3
},
"_links": {
"self": {"href": "/mentions/?feed=https%3A%2F%2Fexample.com%2Fsocial.org", "method": "GET"}
@ -299,8 +445,6 @@ curl -G "http://localhost:8080/mentions/" --data-urlencode "feed=https://example
}
```
The `version` in the `meta` field is a unique identifier for the current state of mentions for the given feed. You can use it to check if there are new mentions since your last request.
### Get reactions
`/reactions/?feed={url feed}` - Get all reactions received by posts from a given feed. A reaction is a special post with a `:MOOD:` property and `:REPLY_TO:` pointing to the reacted post. Results are ordered from most recent to oldest.
@ -336,8 +480,7 @@ curl -G "http://localhost:8080/reactions/" --data-urlencode "feed=https://exampl
],
"meta": {
"feed": "https://example.com/social.org",
"total": 3,
"version": "123"
"total": 3
},
"_links": {
"self": {"href": "/reactions/?feed=https%3A%2F%2Fexample.com%2Fsocial.org", "method": "GET"}
@ -352,8 +495,6 @@ The response includes:
To extract the author's feed from the `post` field, simply take the part before the `#` character. For example, from `https://alice.org/social.org#2025-02-05T13:15:00+0100`, the author is `https://alice.org/social.org`.
The `version` in the `meta` field is a unique identifier for the current state of reactions for the given feed. You can use it to check if there are new reactions since your last request.
**Note:** According to Org Social specification, reactions are posts with:
- `:REPLY_TO:` property pointing to the reacted post
- `:MOOD:` property containing the emoji
@ -391,8 +532,7 @@ curl -G "http://localhost:8080/replies-to/" --data-urlencode "feed=https://examp
],
"meta": {
"feed": "https://example.com/social.org",
"total": 3,
"version": "123"
"total": 3
},
"_links": {
"self": {"href": "/replies-to/?feed=https%3A%2F%2Fexample.com%2Fsocial.org", "method": "GET"}
@ -406,10 +546,161 @@ The response includes:
To extract the author's feed from the `post` field, simply take the part before the `#` character. For example, from `https://alice.org/social.org#2025-02-05T13:15:00+0100`, the author is `https://alice.org/social.org`.
The `version` in the `meta` field is a unique identifier for the current state of replies for the given feed. You can use it to check if there are new replies since your last request.
**Note:** This endpoint shows direct replies to your posts. To see the full conversation thread of a specific post, use the `/replies/?post={post_url}` endpoint instead.
### Get boosts
`/boosts/?post={url post}` - Get all boosts (reposts/shares) for a specific post. A boost is when someone shares your post on their timeline using the `:INCLUDE:` property. Results are ordered from most recent to oldest.
```sh
# URL must be encoded when passed as query parameter
curl "http://localhost:8080/boosts/?post=https%3A%2F%2Fexample.com%2Fsocial.org%232025-02-05T10%3A00%3A00%2B0100"
# Or use curl's --data-urlencode for automatic encoding:
curl -G "http://localhost:8080/boosts/" --data-urlencode "post=https://example.com/social.org#2025-02-05T10:00:00+0100"
```
```json
{
"type": "Success",
"errors": [],
"data": [
"https://alice.org/social.org#2025-02-05T14:00:00+0100",
"https://bob.org/social.org#2025-02-05T15:30:00+0100",
"https://charlie.org/social.org#2025-02-05T16:45:00+0100"
],
"meta": {
"post": "https://example.com/social.org#2025-02-05T10:00:00+0100",
"total": 3
},
"_links": {
"self": {"href": "/boosts/?post=https%3A%2F%2Fexample.com%2Fsocial.org%232025-02-05T10%3A00%3A00%2B0100", "method": "GET"}
}
}
```
The response includes:
- A list of boost post URLs (format: `{booster_feed}#{timestamp}`)
To extract the booster's feed from each post URL, simply take the part before the `#` character. For example, from `https://alice.org/social.org#2025-02-05T14:00:00+0100`, the booster is `https://alice.org/social.org`.
**Note:** According to Org Social specification, boosts are posts with the `:INCLUDE:` property pointing to the boosted post.
### Get interactions (all-in-one)
`/interactions/?post={url post}` - Get all interactions for a specific post in a single request. This endpoint consolidates reactions, replies, and boosts for optimal performance. Results are ordered from most recent to oldest.
```sh
# URL must be encoded when passed as query parameter
curl "http://localhost:8080/interactions/?post=https%3A%2F%2Fexample.com%2Fsocial.org%232025-02-05T10%3A00%3A00%2B0100"
# Or use curl's --data-urlencode for automatic encoding:
curl -G "http://localhost:8080/interactions/" --data-urlencode "post=https://example.com/social.org#2025-02-05T10:00:00+0100"
```
```json
{
"type": "Success",
"errors": [],
"data": {
"reactions": [
{
"post": "https://alice.org/social.org#2025-02-05T13:15:00+0100",
"emoji": "❤"
},
{
"post": "https://bob.org/social.org#2025-02-05T14:30:00+0100",
"emoji": "🚀"
}
],
"replies": [
"https://charlie.org/social.org#2025-02-05T12:30:00+0100",
"https://diana.org/social.org#2025-02-05T15:00:00+0100"
],
"boosts": [
"https://alice.org/social.org#2025-02-05T14:00:00+0100",
"https://bob.org/social.org#2025-02-05T15:30:00+0100"
]
},
"meta": {
"post": "https://example.com/social.org#2025-02-05T10:00:00+0100",
"total_reactions": 2,
"total_replies": 2,
"total_boosts": 2,
"parentChain": [
"https://original.org/social.org#2025-02-04T09:00:00+0100",
"https://parent.org/social.org#2025-02-05T08:00:00+0100"
]
},
"_links": {
"self": {"href": "/interactions/?post=https%3A%2F%2Fexample.com%2Fsocial.org%232025-02-05T10%3A00%3A00%2B0100", "method": "GET"},
"reactions": {"href": "/reactions/?feed=https%3A%2F%2Fexample.com%2Fsocial.org", "method": "GET"},
"replies": {"href": "/replies/?post=https%3A%2F%2Fexample.com%2Fsocial.org%232025-02-05T10%3A00%3A00%2B0100", "method": "GET"},
"boosts": {"href": "/boosts/?post=https%3A%2F%2Fexample.com%2Fsocial.org%232025-02-05T10%3A00%3A00%2B0100", "method": "GET"}
}
}
```
The response includes:
- `reactions`: Array of reaction objects with `post` (URL) and `emoji`
- `replies`: Array of reply post URLs (excludes reactions - posts without mood)
- `boosts`: Array of boost post URLs
- `parentChain`: Array of parent post URLs from oldest to most recent (empty if post is root)
**Note:** This endpoint is optimized for displaying post details in a single request. It excludes nested replies (use `/replies/?post={post_url}` for the full thread tree) and only includes direct replies to maintain simplicity and performance.
The `parentChain` allows you to reconstruct the conversation context by showing all parent posts from the original root post up to the immediate parent. If the post is a reply to another post, you'll get the full chain; if it's a root post, the array will be empty.
**Use cases:**
- Display a post with all its interactions in one request
- Show conversation context with parent chain
- Optimize mobile/web apps by reducing HTTP requests
- Get post engagement metrics (reactions, replies, boosts count)
### Get raw feed content
`/feed-content/?feed={url feed}` - Get the raw content of an Org Social feed file. This endpoint fetches and returns the original `.org` file content from the specified feed URL.
```sh
# URL must be encoded when passed as query parameter
curl "http://localhost:8080/feed-content/?feed=https%3A%2F%2Fexample.com%2Fsocial.org"
# Or use curl's --data-urlencode for automatic encoding:
curl -G "http://localhost:8080/feed-content/" --data-urlencode "feed=https://example.com/social.org"
```
```json
{
"type": "Success",
"errors": [],
"data": {
"content": "#+TITLE: My Social Feed\n#+AUTHOR: John Doe\n\n* 2025-02-05T10:00:00+0100\n:PROPERTIES:\n:ID: 2025-02-05T10:00:00+0100\n:END:\n\nHello, world! This is my first post.\n\n* 2025-02-05T12:30:00+0100\n:PROPERTIES:\n:ID: 2025-02-05T12:30:00+0100\n:REPLY_TO: https://alice.org/social.org#2025-02-05T10:15:00+0100\n:END:\n\nThis is a reply to Alice's post.\n"
},
"_links": {
"self": {"href": "/feed-content/?feed=https%3A%2F%2Fexample.com%2Fsocial.org", "method": "GET"}
}
}
```
The response includes:
- `content`: The raw text content of the feed file (Org Mode format)
**Use cases:**
- Debug feed format issues
- Parse feeds locally in client applications
- Backup or archive feed content
- Analyze feed structure and properties
- Validate Org Social format compliance
**Error handling:**
- Returns 400 if the `feed` parameter is missing or invalid
- Returns 404 if the feed is not registered in the relay
- Returns 502 if the feed URL cannot be fetched (network error, server down, etc.)
**Note:**
- This endpoint fetches the feed content directly from the source URL in real-time and does not use cached data.
- The content is returned exactly as stored in the original `.org` file, preserving all formatting, whitespace, and Org Mode properties.
### Get replies/threads
`/replies/?post={url post}` - Get replies for a given post. This will return a tree structure with all the replies to posts in the given feed. If you want to see the entire tree, you must use the meta `parent` as a `post`.
@ -485,8 +776,11 @@ curl -G "http://localhost:8080/replies/" --data-urlencode "post=https://foo.org/
}
],
"meta": {
"parent": "https://moo.org/social.org#2025-02-03T23:05:00+0100",
"version": "123"
"parent": "https://foo.org/social.org#2025-02-03T23:05:00+0100",
"parentChain": [
"https://root.org/social.org#2025-02-01T10:00:00+0100",
"https://foo.org/social.org#2025-02-03T23:05:00+0100"
]
},
"_links": {
"self": {"href": "/replies/?post=https%3A%2F%2Ffoo.org%2Fsocial.org%232025-02-03T23%3A05%3A00%2B0100", "method": "GET"}
@ -494,7 +788,10 @@ curl -G "http://localhost:8080/replies/" --data-urlencode "post=https://foo.org/
}
```
The `version` in the `meta` field is a unique identifier for the current state of replies for the given post. You can use it to check if there are new replies since your last request.
Each node in the tree includes:
- `post`: The post URL
- `children`: Array of direct reply nodes (recursive structure)
- `moods`: Array of emoji reactions with their posts
### Search
@ -520,7 +817,6 @@ Optional parameters:
"..."
],
"meta": {
"version": "123",
"query": "example",
"total": 150,
"page": 1,
@ -536,8 +832,6 @@ Optional parameters:
}
```
The `version` in the `meta` field is a unique identifier for the current state of the search index. You can use it to check if there are new results since your last request.
### List groups
`/groups/` - List all groups from the relay.
@ -623,8 +917,7 @@ curl http://localhost:8080/groups/emacs/
"https://alice.org/social.org",
"https://bob.org/social.org",
"https://charlie.org/social.org"
],
"version": "123"
]
},
"_links": {
"self": {"href": "/groups/emacs/", "method": "GET"},
@ -633,8 +926,6 @@ curl http://localhost:8080/groups/emacs/
}
```
The `version` in the `meta` field is a unique identifier for the current state of messages in the group. You can use it to check if there are new messages since your last request.
### List polls
`/polls/` - List all polls from the relay. Results are ordered from most recent to oldest.
@ -653,8 +944,7 @@ curl http://localhost:8080/polls/
"https://baz.org/social.org#2025-02-05T08:30:00+0100"
],
"meta": {
"total": 3,
"version": "123"
"total": 3
},
"_links": {
"self": {"href": "/polls/", "method": "GET"},
@ -663,7 +953,47 @@ curl http://localhost:8080/polls/
}
```
The `version` in the `meta` field is a unique identifier for the current state of polls. You can use it to check if there are new polls since your last request.
### Get profile
`/profile/?feed={url feed}` - Get profile information for a given feed, including the list of feeds that follow it (followers).
```sh
# URL must be encoded when passed as query parameter
curl "http://localhost:8080/profile/?feed=https%3A%2F%2Fexample.com%2Fsocial.org"
# Or use curl's --data-urlencode for automatic encoding:
curl -G "http://localhost:8080/profile/" --data-urlencode "feed=https://example.com/social.org"
```
```json
{
"type": "Success",
"errors": [],
"data": {
"feed": "https://example.com/social.org",
"followers": [
"https://alice.org/social.org",
"https://bob.org/social.org",
"https://charlie.org/social.org"
]
},
"meta": {
"feed": "https://example.com/social.org",
"total_followers": 3
},
"_links": {
"self": {"href": "/profile/?feed=https%3A%2F%2Fexample.com%2Fsocial.org", "method": "GET"}
}
}
```
The response includes:
- `feed`: The queried feed URL
- `followers`: List of feed URLs that follow this feed (i.e. feeds that have included this feed in their following list)
**Error handling:**
- Returns 400 if the `feed` parameter is missing or invalid
- Returns 404 if the feed is not registered in the relay
### Get poll votes
@ -708,8 +1038,7 @@ curl -G "http://localhost:8080/polls/votes/" --data-urlencode "post=https://foo.
],
"meta": {
"poll": "https://foo.org/social.org#2025-02-03T23:05:00+0100",
"total_votes": 4,
"version": "123"
"total_votes": 4
},
"_links": {
"self": {"href": "/polls/votes/?post=https%3A%2F%2Ffoo.org%2Fsocial.org%232025-02-03T23%3A05%3A00%2B0100", "method": "GET"},
@ -718,7 +1047,182 @@ curl -G "http://localhost:8080/polls/votes/" --data-urlencode "post=https://foo.
}
```
The `version` in the `meta` field is a unique identifier for the current state of votes for the given poll. You can use it to check if there are new votes since your last request.
### Get statistics
`/stats/` - Get aggregated statistics about the activity registered by the Relay. Data is grouped by year and month, plus a block of global counters. The month of each post is determined by its ID (RFC 3339 timestamp), not by when the Relay discovered it.
```sh
curl "http://localhost:8080/stats/"
```
```json
{
"type": "Success",
"errors": [],
"data": {
"years": {
"2025": {
"11": {
"active_accounts": 42,
"total_posts": 512,
"posts": 300,
"replies": 120,
"boosts": 40,
"reactions": 35,
"group_messages": 15,
"polls": 7
},
"12": {
"active_accounts": 51,
"total_posts": 640,
"posts": 380,
"replies": 150,
"boosts": 52,
"reactions": 41,
"group_messages": 22,
"polls": 4
}
},
"2026": {
"01": {
"active_accounts": 58,
"total_posts": 701,
"posts": 410,
"replies": 170,
"boosts": 60,
"reactions": 48,
"group_messages": 25,
"polls": 9
}
}
},
"global": {
"registered_feeds": 128,
"total_accounts": 115,
"total_posts": 4210,
"total_follows": 340,
"active_groups": 6
}
},
"meta": {
"generated_at": "2026-01-15T10:30:00+00:00"
},
"_links": {
"self": {"href": "/stats/", "method": "GET"},
"feeds": {"href": "/feeds/", "method": "GET"}
}
}
```
Counter definitions per month:
- `active_accounts`: Number of distinct accounts that published at least one post (of any kind) during that month.
- `total_posts`: Every post registered that month, regardless of its type.
- `posts`: Original posts, meaning posts that are neither a reply (`:REPLY_TO:`) nor a boost (`:INCLUDE:`).
- `replies`: Posts with the `:REPLY_TO:` property that contain text content. Reactions are excluded.
- `boosts`: Posts with the `:INCLUDE:` property.
- `reactions`: Replies with a `:MOOD:` property and no text content.
- `group_messages`: Posts published in any group (`:GROUP:` property).
- `polls`: Polls created that month (posts with `:POLL_END:`). Poll closings are not counted.
Global counters:
- `registered_feeds`: Total number of feeds currently registered in the Relay.
- `total_accounts`: Total number of accounts (profiles) known by the Relay.
- `total_posts`: Total number of posts registered, all time.
- `total_follows`: Total number of follow relationships between accounts.
- `active_groups`: Number of groups with at least one message.
**Note:** Counters are not mutually exclusive: a reply inside a group counts in both `replies` and `group_messages`, and a poll also counts in `posts`. Only `posts`, `replies`, `reactions` and `boosts` are disjoint between them, and together they add up to `total_posts`. Months with no activity are omitted. Like the rest of the endpoints, the response is cached and refreshed after each feed scan.
### Bridges (follow ActivityPub accounts and RSS feeds)
Bridges expose external accounts as **virtual social.org feeds** served by the relay. Any Org Social client can follow them with a plain `#+FOLLOW:` line, no client changes needed.
Registration is implicit: the first request of an unknown account fetches it from the origin and stores it. Active bridges are refreshed every 15 minutes; bridges nobody has requested for 90 days are removed automatically (and re-created on the next request).
#### ActivityPub bridge
`/bridge/activitypub/@{user}@{instance}/` - A Mastodon (or any ActivityPub) account as a virtual social.org feed (plain text).
```sh
curl http://localhost:8080/bridge/activitypub/@Mastodon@mastodon.social/
```
```org
#+TITLE: Mastodon
#+NICK: Mastodon
#+DESCRIPTION: Our mission is to connect the world...
#+AVATAR: https://files.mastodon.social/accounts/avatars/.../avatar.png
#+LINK: https://mastodon.social/@Mastodon
* Posts
** 2026-03-27T16:27:02+00:00
:PROPERTIES:
:LANG: en
:TAGS: fossback mastodon opensourcedesign ux
:END:
Following on from yesterday's blog post...
```
To follow the account, users add to their `social.org`:
```org
#+FOLLOW: https://relay.org-social.org/bridge/activitypub/@Mastodon@mastodon.social/
```
Notes:
- Handles are case-insensitive; non-canonical spellings redirect (301) to the lowercase canonical URL.
- Only public top-level posts are bridged (no replies, no boosts), like Mastodon's public RSS.
- Content warnings become a `CW:` first line; media attachments become Org links; hashtags become `:TAGS:`.
- Instances running in *authorized fetch* mode (signed requests required) cannot be bridged.
`/bridge/activitypub/` (without handle) lists the accounts already bridged by this relay.
#### RSS/Atom bridge
`/bridge/rss/?url={feed_url}` - An RSS or Atom feed as a virtual social.org feed (plain text). The `url` parameter must be URL-encoded.
```sh
curl -G http://localhost:8080/bridge/rss/ --data-urlencode "url=https://xkcd.com/rss.xml"
```
```org
#+TITLE: xkcd.com
#+NICK: xkcd_com
#+DESCRIPTION: xkcd.com: A webcomic of romance and math humor.
#+LINK: https://xkcd.com/
* Posts
** 2026-07-08T04:00:00+00:00
:PROPERTIES:
:END:
*** Airport Meeting
[[https://imgs.xkcd.com/comics/airport_meeting.png][Although it was a setback for physics...]]
[[https://xkcd.com/3269/]]
```
To follow the feed:
```org
#+FOLLOW: https://relay.org-social.org/bridge/rss/?url=https%3A%2F%2Fxkcd.com%2Frss.xml
```
Notes:
- Entry titles become `***` sub-headings inside the post, followed by the converted content and a link to the original article.
- Entries without a publication date are skipped; entries sharing the same date get IDs shifted forward one second so none is lost.
`/bridge/rss/` (without `url`) lists the feeds already bridged by this relay.
#### Bridged feeds and the rest of the API
Bridged accounts are stored as regular profiles, so the existing endpoints (`/profile/`, `/search/`, `/rss.xml?feed=...`, ...) work with the bridge URL as the `feed` parameter. Bridged posts never trigger Webmentions nor notifications: the relay only mirrors external content.
## Groups Configuration
@ -760,9 +1264,33 @@ Once configured, users can:
The groups endpoints will only be available when groups are configured via the `GROUPS` environment variable.
## RSS Feed
Org Social Relay provides an RSS feed of different types of content.
- The latest posts scanned from all registered feeds.
```sh
http://localhost:8080/rss.xml
```
- By tag.
```sh
curl http://localhost:8080/rss.xml?tag=emacs
```
- By author feed.
```sh
curl http://localhost:8080/rss.xml?feed=https%3A%2F%2Forg-social.org%2Fsocial.org
```
This RSS feed can be used in RSS readers to stay updated with new posts from the relay, but is limited to the latest 200 posts.
## Technical information
You can find the public Relay list in `https://cdn.jsdelivr.net/gh/tanrax/org-social/org-social-relay-list.txt`.
The relay list is stored locally in `relay-list.txt`.
### Crons
@ -783,3 +1311,19 @@ Every day at midnight, Relay analyzes the feeds of all registered users to disco
#### Cleanup stale feeds
Every 3 days at 2 AM, Relay automatically removes feeds that haven't been successfully fetched (HTTP 200) in the last 3 days. This keeps the relay efficient by removing inactive or dead feeds.
#### Refresh bridges
Every 15 minutes, Relay refetches the bridged ActivityPub accounts and RSS feeds that have been requested in the last 7 days. Dormant bridges are refreshed inline on their next request instead.
#### Cleanup stale bridges
Every 3 days at 3 AM, Relay removes bridged accounts that nobody has requested in the last 90 days, together with their stored profile and posts. They are re-created automatically if someone requests them again.
## Contributing
Contributions are welcome! Please see the [contribution guidelines](https://git.andros.dev/andros/contribute) for instructions on how to submit issues or pull requests.
## License
Org Social is free software, released under the [GPLv3 license](https://www.gnu.org/licenses/gpl-3.0.html).

0
app/boosts/__init__.py Normal file
View file

6
app/boosts/apps.py Normal file
View file

@ -0,0 +1,6 @@
from django.apps import AppConfig
class BoostsConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "app.boosts"

227
app/boosts/tests.py Normal file
View file

@ -0,0 +1,227 @@
from django.test import TestCase
from rest_framework.test import APIClient
from rest_framework import status
from app.feeds.models import Profile, Post
class BoostsViewTest(TestCase):
"""Test cases for the BoostsView API using Given/When/Then structure."""
def setUp(self):
self.client = APIClient()
self.boosts_url = "/boosts/"
# Create test profiles
self.profile1 = Profile.objects.create(
feed="https://example.com/social.org",
title="Example Profile",
nick="example_user",
description="Test profile 1",
)
self.profile2 = Profile.objects.create(
feed="https://alice.com/social.org",
title="Alice Profile",
nick="alice",
description="Alice's profile",
)
self.profile3 = Profile.objects.create(
feed="https://bob.com/social.org",
title="Bob Profile",
nick="bob",
description="Bob's profile",
)
self.profile4 = Profile.objects.create(
feed="https://charlie.com/social.org",
title="Charlie Profile",
nick="charlie",
description="Charlie's profile",
)
# Create original post
self.original_post = Post.objects.create(
profile=self.profile1,
post_id="2025-02-05T10:00:00+0100",
content="This is an amazing discovery!",
)
# Create boosts of the original post
self.boost1 = Post.objects.create(
profile=self.profile2,
post_id="2025-02-05T14:00:00+0100",
content="Guys, you have to see this!",
include=f"{self.profile1.feed}#{self.original_post.post_id}",
)
self.boost2 = Post.objects.create(
profile=self.profile3,
post_id="2025-02-05T15:30:00+0100",
content="", # Simple boost without comment
include=f"{self.profile1.feed}#{self.original_post.post_id}",
)
self.boost3 = Post.objects.create(
profile=self.profile4,
post_id="2025-02-05T16:45:00+0100",
content="", # Another simple boost
include=f"{self.profile1.feed}#{self.original_post.post_id}",
)
# Create another post without boosts
self.post_without_boosts = Post.objects.create(
profile=self.profile1,
post_id="2025-02-05T11:00:00+0100",
content="Another post without boosts",
)
# Create a post that boosts a different post
self.other_post = Post.objects.create(
profile=self.profile2,
post_id="2025-02-05T09:00:00+0100",
content="Different post",
)
self.boost_other = Post.objects.create(
profile=self.profile3,
post_id="2025-02-05T17:00:00+0100",
content="Boosting different post",
include=f"{self.profile2.feed}#{self.other_post.post_id}",
)
def test_get_boosts_success(self):
"""Test GET /boosts/?post=<post_url> returns list of boosts."""
# Given: A post with boosts exists
post_url = f"{self.profile1.feed}#{self.original_post.post_id}"
# When: We request boosts for the post
response = self.client.get(self.boosts_url, {"post": post_url})
# Then: We should get boosts successfully
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["type"], "Success")
self.assertEqual(response.data["errors"], [])
# Then: Response should contain list of boost post URLs
data = response.data["data"]
self.assertIsInstance(data, list)
self.assertEqual(len(data), 3) # 3 boosts
# Then: Boosts should be ordered by post_id (most recent first)
expected_boosts = [
f"{self.profile4.feed}#{self.boost3.post_id}",
f"{self.profile3.feed}#{self.boost2.post_id}",
f"{self.profile2.feed}#{self.boost1.post_id}",
]
self.assertEqual(data, expected_boosts)
# Then: Meta should contain post URL and total count
meta = response.data["meta"]
self.assertEqual(meta["post"], post_url)
self.assertEqual(meta["total"], 3)
# Then: Links should contain self reference
links = response.data["_links"]
self.assertIn("self", links)
self.assertIn("/boosts/", links["self"]["href"])
def test_get_boosts_no_boosts(self):
"""Test GET /boosts/?post=<post_url> when post has no boosts."""
# Given: A post without boosts exists
post_url = f"{self.profile1.feed}#{self.post_without_boosts.post_id}"
# When: We request boosts for the post
response = self.client.get(self.boosts_url, {"post": post_url})
# Then: We should get successful response with empty list
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["type"], "Success")
data = response.data["data"]
self.assertEqual(len(data), 0)
self.assertEqual(response.data["meta"]["total"], 0)
def test_get_boosts_missing_post_parameter(self):
"""Test GET /boosts/ without post parameter returns error."""
# Given: No post parameter provided
# When: We request boosts without post parameter
response = self.client.get(self.boosts_url)
# Then: We should get a 400 error
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(response.data["type"], "Error")
self.assertIn("'post' parameter is required", response.data["errors"])
def test_get_boosts_invalid_post_url_format(self):
"""Test GET /boosts/?post=<invalid_url> returns error."""
# Given: An invalid post URL (missing #)
invalid_url = "https://example.com/social.org"
# When: We request boosts with invalid URL
response = self.client.get(self.boosts_url, {"post": invalid_url})
# Then: We should get a 400 error
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(response.data["type"], "Error")
self.assertIn("Invalid post URL format", response.data["errors"][0])
def test_get_boosts_post_not_found(self):
"""Test GET /boosts/?post=<nonexistent_post> returns 404."""
# Given: A non-existent post URL
post_url = "https://nonexistent.com/social.org#2025-01-01T00:00:00+00:00"
# When: We request boosts for non-existent post
response = self.client.get(self.boosts_url, {"post": post_url})
# Then: We should get a 404 error
self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)
self.assertEqual(response.data["type"], "Error")
self.assertIn("Post not found", response.data["errors"])
def test_get_boosts_profile_not_found(self):
"""Test GET /boosts/?post=<url_with_unknown_profile> returns 404."""
# Given: A URL with unknown profile
post_url = "https://unknown-profile.com/social.org#2025-01-01T00:00:00+00:00"
# When: We request boosts
response = self.client.get(self.boosts_url, {"post": post_url})
# Then: We should get a 404 error
self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)
self.assertEqual(response.data["type"], "Error")
self.assertIn("Post not found", response.data["errors"])
def test_boosts_isolated_by_post(self):
"""Test that boosts are correctly isolated by post."""
# Given: Multiple posts with boosts exist
post1_url = f"{self.profile1.feed}#{self.original_post.post_id}"
post2_url = f"{self.profile2.feed}#{self.other_post.post_id}"
# When: We request boosts for post 1
response1 = self.client.get(self.boosts_url, {"post": post1_url})
# Then: We should only get boosts for post 1
self.assertEqual(len(response1.data["data"]), 3)
# When: We request boosts for post 2
response2 = self.client.get(self.boosts_url, {"post": post2_url})
# Then: We should only get boosts for post 2
self.assertEqual(len(response2.data["data"]), 1)
self.assertIn(
f"{self.profile3.feed}#{self.boost_other.post_id}",
response2.data["data"],
)
def test_boosts_caching(self):
"""Test that boosts responses are cached."""
# Given: A post with boosts
post_url = f"{self.profile1.feed}#{self.original_post.post_id}"
# When: We request boosts twice
response1 = self.client.get(self.boosts_url, {"post": post_url})
response2 = self.client.get(self.boosts_url, {"post": post_url})
# Then: Both responses should be identical
self.assertEqual(response1.data, response2.data)
self.assertEqual(response1.status_code, status.HTTP_200_OK)
self.assertEqual(response2.status_code, status.HTTP_200_OK)

8
app/boosts/urls.py Normal file
View file

@ -0,0 +1,8 @@
from django.urls import path
from .views import BoostsView
app_name = "boosts"
urlpatterns = [
path("", BoostsView.as_view(), name="boosts"),
]

22
app/boosts/utils.py Normal file
View file

@ -0,0 +1,22 @@
"""
Utility functions for boosts
"""
from app.feeds.models import Post
def get_boosts_for_post(post_url: str):
"""
Get all boosts for a specific post.
Args:
post_url: The post URL in format feed_url#post_id
Returns:
QuerySet of Post objects that boost the given post
"""
return (
Post.objects.filter(include=post_url)
.select_related("profile")
.order_by("-post_id")
)

97
app/boosts/views.py Normal file
View file

@ -0,0 +1,97 @@
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework import status
from django.core.cache import cache
import logging
from app.feeds.models import Post, Profile
logger = logging.getLogger(__name__)
class BoostsView(APIView):
"""Get boosts for a specific post"""
def get(self, request):
post_url = request.query_params.get("post")
if not post_url:
return Response(
{
"type": "Error",
"errors": ["'post' parameter is required"],
"data": None,
},
status=status.HTTP_400_BAD_REQUEST,
)
# Parse the post URL format: https://feed.com/social.org#post_id
try:
if "#" not in post_url:
raise ValueError("Invalid post URL format")
feed_url, post_id = post_url.split("#", 1)
except ValueError:
return Response(
{
"type": "Error",
"errors": ["Invalid post URL format. Expected: feed_url#post_id"],
"data": None,
},
status=status.HTTP_400_BAD_REQUEST,
)
cache_key = f"boosts_{feed_url}_{post_id}"
cached_response = cache.get(cache_key)
if cached_response is not None:
return Response(cached_response, status=status.HTTP_200_OK)
# Find the original post
try:
profile = Profile.objects.get(feed=feed_url)
Post.objects.get(profile=profile, post_id=post_id)
except (Profile.DoesNotExist, Post.DoesNotExist):
return Response(
{
"type": "Error",
"errors": ["Post not found"],
"data": None,
},
status=status.HTTP_404_NOT_FOUND,
)
# Get all boosts for this post
# A boost is a post with include property pointing to this post
original_post_url = f"{feed_url}#{post_id}"
boosts = (
Post.objects.filter(include=original_post_url)
.select_related("profile")
.order_by("-post_id")
)
# Build data according to README spec (simple list of post URLs)
boosts_data = [f"{boost.profile.feed}#{boost.post_id}" for boost in boosts]
# URL encode the post_url for the self link
from urllib.parse import quote
encoded_post_url = quote(post_url, safe="")
response_data = {
"type": "Success",
"errors": [],
"data": boosts_data,
"meta": {
"post": original_post_url,
"total": len(boosts_data),
},
"_links": {
"self": {"href": f"/boosts/?post={encoded_post_url}", "method": "GET"}
},
}
# Cache permanently (will be cleared by scan_feeds task)
cache.set(cache_key, response_data, None)
return Response(response_data, status=status.HTTP_200_OK)

0
app/bridge/__init__.py Normal file
View file

215
app/bridge/activitypub.py Normal file
View file

@ -0,0 +1,215 @@
"""
ActivityPub fetching for the bridge: WebFinger resolution, actor document
and outbox pagination, normalized into a bridge-agnostic structure.
Only public top-level notes are bridged: replies and boosts (Announce)
are skipped, mirroring what Mastodon exposes in its public RSS feeds.
Instances that require signed fetches (authorized fetch mode) will
answer 401/403 and the account cannot be bridged.
"""
import json
import logging
import re
from urllib.parse import quote
from dateutil import parser as date_parser
from .fetching import BridgeError, safe_get
from .html_to_org import html_to_org, html_to_text
logger = logging.getLogger(__name__)
HANDLE_RE = re.compile(r"^[a-z0-9._-]{1,64}@[a-z0-9-]+(\.[a-z0-9-]+)+$")
ACTIVITYPUB_ACCEPT = (
"application/activity+json, "
'application/ld+json; profile="https://www.w3.org/ns/activitystreams"'
)
MAX_POSTS = 40
MAX_OUTBOX_PAGES = 3
def normalize_handle(raw_handle):
"""
Normalize a user@instance handle: lowercase and without the optional
leading @. Returns None if the result is not a valid handle.
"""
handle = (raw_handle or "").strip().lstrip("@").lower()
if not HANDLE_RE.match(handle):
return None
return handle
def _get_json(url, description):
try:
return json.loads(safe_get(url, accept=ACTIVITYPUB_ACCEPT))
except json.JSONDecodeError as e:
raise BridgeError(f"Invalid JSON in {description} at {url}") from e
def _resolve_actor_url(user, instance):
"""Resolve the actor URL of user@instance through WebFinger."""
resource = quote(f"acct:{user}@{instance}", safe="")
webfinger_url = f"https://{instance}/.well-known/webfinger?resource={resource}"
try:
document = json.loads(safe_get(webfinger_url, accept="application/jrd+json"))
except json.JSONDecodeError as e:
raise BridgeError(f"Invalid WebFinger response from {instance}") from e
for link in document.get("links", []):
if not isinstance(link, dict) or link.get("rel") != "self":
continue
if "activity+json" in (link.get("type") or "") and link.get("href"):
return link["href"]
for link in document.get("links", []):
if isinstance(link, dict) and link.get("rel") == "self" and link.get("href"):
return link["href"]
raise BridgeError(f"No ActivityPub actor found for {user}@{instance}")
def _format_post_id(published):
"""Convert an ActivityPub published date to an Org Social post ID."""
parsed = date_parser.parse(published)
if parsed.tzinfo is None:
return None
from datetime import timezone as dt_timezone
return parsed.astimezone(dt_timezone.utc).strftime("%Y-%m-%dT%H:%M:%S+00:00")
def _note_to_post(note):
"""Normalize an ActivityPub Note object into a bridge post dict."""
published = note.get("published")
if not published:
return None
try:
post_id = _format_post_id(published)
except (ValueError, OverflowError):
logger.warning(f"Skipping note with unparseable date: {published}")
return None
if post_id is None:
return None
parts = []
summary = note.get("summary")
if summary:
parts.append(f"CW: {html_to_text(summary)}")
body = html_to_org(note.get("content") or "")
if body:
parts.append(body)
attachment_links = []
for attachment in note.get("attachment", []):
if isinstance(attachment, dict) and attachment.get("url"):
attachment_links.append(f"[[{attachment['url']}]]")
if attachment_links:
parts.append("\n".join(attachment_links))
tags = []
for tag in note.get("tag", []):
if isinstance(tag, dict) and tag.get("type") == "Hashtag" and tag.get("name"):
tags.append(tag["name"].lstrip("#"))
language = ""
content_map = note.get("contentMap")
if isinstance(content_map, dict) and content_map:
language = next(iter(content_map.keys()), "")
return {
"id": post_id,
"content": "\n\n".join(parts),
"tags": " ".join(tags),
"language": language,
}
def _collect_outbox_posts(outbox_url):
"""Walk the outbox pages collecting public top-level notes."""
document = _get_json(outbox_url, "outbox")
if "orderedItems" in document:
page = document
else:
first = document.get("first")
if isinstance(first, str):
page = _get_json(first, "outbox page")
elif isinstance(first, dict):
page = first
else:
return []
posts = []
seen_ids = set()
pages_read = 0
while page and pages_read < MAX_OUTBOX_PAGES and len(posts) < MAX_POSTS:
pages_read += 1
for item in page.get("orderedItems", []):
if len(posts) >= MAX_POSTS:
break
if not isinstance(item, dict) or item.get("type") != "Create":
continue
note = item.get("object")
if not isinstance(note, dict) or note.get("type") != "Note":
continue
if note.get("inReplyTo"):
continue
post = _note_to_post(note)
if post is None or post["id"] in seen_ids:
continue
seen_ids.add(post["id"])
posts.append(post)
next_url = page.get("next")
if isinstance(next_url, str) and len(posts) < MAX_POSTS:
page = _get_json(next_url, "outbox page")
else:
page = None
return posts
def _actor_avatar(actor):
icon = actor.get("icon")
if isinstance(icon, list):
icon = icon[0] if icon else None
if isinstance(icon, dict):
return icon.get("url") or ""
if isinstance(icon, str):
return icon
return ""
def fetch_account(user, instance):
"""
Fetch an ActivityPub account and normalize it for the bridge.
Returns a dict with actor_url, outbox_url, metadata and posts.
Raises BridgeError when the account cannot be fetched.
"""
actor_url = _resolve_actor_url(user, instance)
actor = _get_json(actor_url, "actor document")
outbox_url = actor.get("outbox")
nick = actor.get("preferredUsername") or user
metadata = {
"title": actor.get("name") or f"{nick}@{instance}",
"nick": nick,
"description": html_to_text(actor.get("summary") or ""),
"avatar": _actor_avatar(actor),
"link": actor.get("url") or actor_url,
}
posts = []
if isinstance(outbox_url, str) and outbox_url:
posts = _collect_outbox_posts(outbox_url)
else:
logger.warning(f"Actor {actor_url} has no outbox")
outbox_url = ""
return {
"actor_url": actor_url,
"outbox_url": outbox_url,
"metadata": metadata,
"posts": posts,
}

6
app/bridge/apps.py Normal file
View file

@ -0,0 +1,6 @@
from django.apps import AppConfig
class BridgeConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "app.bridge"

52
app/bridge/fetching.py Normal file
View file

@ -0,0 +1,52 @@
"""
Shared HTTP fetching helpers for the bridge app.
All bridge fetches go through safe_get(), which enforces the same SSRF
protections used for Webmentions, plus a timeout and a response size cap.
"""
import logging
import requests
from app.feeds.webmentions import is_safe_endpoint
logger = logging.getLogger(__name__)
BRIDGE_TIMEOUT = 10
BRIDGE_USER_AGENT = "Org-Social-Relay-Bridge (+https://relay.org-social.org/)"
# Maximum bytes read from any remote response
BRIDGE_MAX_BYTES = 5 * 1024 * 1024
class BridgeError(Exception):
"""A bridge source could not be fetched or understood."""
def safe_get(url, accept=None):
"""
GET a remote URL with SSRF protection, timeout and size cap.
Returns the raw bytes of the response body.
Raises BridgeError on unsafe URLs, network errors or non-2xx codes.
"""
if not is_safe_endpoint(url):
raise BridgeError(f"URL is not allowed: {url}")
headers = {"User-Agent": BRIDGE_USER_AGENT}
if accept:
headers["Accept"] = accept
try:
response = requests.get(
url, timeout=BRIDGE_TIMEOUT, headers=headers, stream=True
)
except requests.RequestException as e:
raise BridgeError(f"Could not reach {url}: {e}") from e
try:
if not (200 <= response.status_code < 300):
raise BridgeError(f"HTTP {response.status_code} from {url}")
return response.raw.read(BRIDGE_MAX_BYTES, decode_content=True)
finally:
response.close()

207
app/bridge/html_to_org.py Normal file
View file

@ -0,0 +1,207 @@
"""
Minimal HTML to Org Mode text converter for bridged content.
Converts the HTML bodies published by ActivityPub servers and RSS feeds
into Org text suitable for a post body inside a virtual social.org file.
Only the common subset is handled (paragraphs, line breaks, links,
emphasis, lists, quotes and preformatted blocks); unknown tags are
stripped, keeping their text.
"""
import re
from html.parser import HTMLParser
# A post body line must never look like an Org Social headline (* or **
# followed by whitespace) because it would break the virtual feed
# structure. Three or more asterisks are legal (post titles).
_HEADLINE_RE = re.compile(r"^(\*{1,2})(\s|$)")
_EMPHASIS_TAGS = {
"strong": "*",
"b": "*",
"em": "/",
"i": "/",
"code": "~",
"tt": "~",
}
_HEADING_TAGS = {"h1", "h2", "h3", "h4", "h5", "h6"}
def escape_org_lines(text):
"""
Prefix a space to any line that would be parsed as an Org Social
headline, so remote content cannot inject fake posts.
"""
lines = []
for line in text.split("\n"):
if _HEADLINE_RE.match(line):
line = " " + line
lines.append(line)
return "\n".join(lines)
class _OrgHTMLConverter(HTMLParser):
def __init__(self):
super().__init__(convert_charrefs=True)
self.blocks = []
self.parts = []
self.anchor_stack = [] # (href, render_as_plain_text, position)
self.emphasis_stack = [] # (marker, position)
self.in_list_item = False
self.pre_depth = 0
self.pre_parts = []
self.invisible_depth = 0
# --- block helpers -------------------------------------------------
def _flush(self):
text = "".join(self.parts)
text = re.sub(r"[ \t]+", " ", text).strip()
self.parts = []
if not text:
return
if self.in_list_item:
self.blocks.append(f"- {text}")
else:
self.blocks.append(text)
def _append_block(self, block):
self._flush()
self.blocks.append(block)
# --- parser events -------------------------------------------------
def handle_starttag(self, tag, attrs):
if self.pre_depth:
if tag == "pre":
self.pre_depth += 1
return
attrs_dict = dict(attrs)
css_classes = (attrs_dict.get("class") or "").lower()
if tag in ("p", "div"):
self._flush()
elif tag == "br":
self.parts.append("\n")
elif tag == "a":
href = (attrs_dict.get("href") or "").strip()
plain = "mention" in css_classes or "hashtag" in css_classes
self.anchor_stack.append((href, plain, len(self.parts)))
elif tag in _EMPHASIS_TAGS:
self.emphasis_stack.append((_EMPHASIS_TAGS[tag], len(self.parts)))
elif tag in ("ul", "ol"):
self._flush()
elif tag == "li":
self._flush()
self.in_list_item = True
elif tag == "blockquote":
self._append_block("#+BEGIN_QUOTE")
elif tag == "pre":
self._flush()
self.pre_depth = 1
self.pre_parts = []
elif tag in _HEADING_TAGS:
self._flush()
self.emphasis_stack.append(("*", len(self.parts)))
elif tag == "img":
src = (attrs_dict.get("src") or "").strip()
if src and "]" not in src:
alt = " ".join((attrs_dict.get("alt") or "").split())
alt = alt.replace("]", ")")
if alt:
self.parts.append(f"[[{src}][{alt}]]")
else:
self.parts.append(f"[[{src}]]")
elif tag == "span" and "invisible" in css_classes:
self.invisible_depth += 1
def handle_endtag(self, tag):
if self.pre_depth:
if tag == "pre":
self.pre_depth -= 1
if self.pre_depth == 0:
code = "".join(self.pre_parts).strip("\n")
self._append_block(f"#+BEGIN_EXAMPLE\n{code}\n#+END_EXAMPLE")
return
if tag == "a" and self.anchor_stack:
href, plain, position = self.anchor_stack.pop()
text = "".join(self.parts[position:]).strip()
if plain or not href or "]" in href or "[[" in text:
# Org links cannot nest: anchors wrapping an image keep
# only the inner image link
rendered = text
elif not text or text == href:
rendered = f"[[{href}]]"
else:
text = text.replace("]", ")")
rendered = f"[[{href}][{text}]]"
self.parts[position:] = [rendered]
elif tag in _EMPHASIS_TAGS and self.emphasis_stack:
marker, position = self.emphasis_stack.pop()
text = "".join(self.parts[position:]).strip()
self.parts[position:] = [f"{marker}{text}{marker}"] if text else []
elif tag in ("p", "div", "ul", "ol"):
self._flush()
elif tag == "li":
self._flush()
self.in_list_item = False
elif tag == "blockquote":
self._append_block("#+END_QUOTE")
elif tag in _HEADING_TAGS and self.emphasis_stack:
marker, position = self.emphasis_stack.pop()
text = "".join(self.parts[position:]).strip()
self.parts[position:] = [f"{marker}{text}{marker}"] if text else []
self._flush()
elif tag == "span" and self.invisible_depth:
self.invisible_depth -= 1
def handle_data(self, data):
if self.pre_depth:
self.pre_parts.append(data)
return
if self.invisible_depth:
return
self.parts.append(data.replace("\n", " "))
def result(self):
self._flush()
# Consecutive list items stay together; everything else is
# separated by a blank line
chunks = []
previous_is_item = False
for block in self.blocks:
is_item = block.startswith("- ")
if chunks and is_item and previous_is_item:
chunks.append("\n")
elif chunks:
chunks.append("\n\n")
chunks.append(block)
previous_is_item = is_item
return "".join(chunks)
def html_to_org(html):
"""
Convert an HTML fragment to Org text. Returns plain text with Org
links/emphasis, paragraphs separated by blank lines, and headline
injection escaped. Never raises on malformed HTML.
"""
if not html:
return ""
converter = _OrgHTMLConverter()
try:
converter.feed(html)
converter.close()
except Exception:
# html.parser is very tolerant; this is a last-resort guard
return escape_org_lines(re.sub(r"<[^>]+>", " ", html).strip())
return escape_org_lines(converter.result())
def html_to_text(html):
"""Convert an HTML fragment to a single line of plain text."""
org = html_to_org(html)
return re.sub(r"\s+", " ", org).strip()

125
app/bridge/migrations/0001_initial.py Normal file
View file

@ -0,0 +1,125 @@
# Generated by Django 6.0.7 on 2026-07-17 13:13
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = [
("feeds", "0011_outgoingwebmention"),
]
operations = [
migrations.CreateModel(
name="BridgedActivityPubAccount",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"handle",
models.CharField(
help_text="Normalized handle in user@instance form (lowercase)",
max_length=255,
unique=True,
),
),
(
"actor_url",
models.URLField(
help_text="ActivityPub actor document URL", max_length=500
),
),
(
"outbox_url",
models.URLField(
blank=True, help_text="ActivityPub outbox URL", max_length=500
),
),
(
"last_refreshed_at",
models.DateTimeField(
blank=True,
help_text="Last successful fetch from the origin",
null=True,
),
),
(
"last_accessed_at",
models.DateTimeField(
help_text="Last time a client requested this virtual feed"
),
),
("created_at", models.DateTimeField(auto_now_add=True)),
(
"profile",
models.OneToOneField(
help_text="Profile that stores the bridged data",
on_delete=django.db.models.deletion.CASCADE,
related_name="activitypub_bridge",
to="feeds.profile",
),
),
],
options={
"ordering": ["handle"],
},
),
migrations.CreateModel(
name="BridgedRssFeed",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"source_url",
models.URLField(
help_text="URL of the RSS/Atom feed",
max_length=500,
unique=True,
),
),
(
"last_refreshed_at",
models.DateTimeField(
blank=True,
help_text="Last successful fetch from the origin",
null=True,
),
),
(
"last_accessed_at",
models.DateTimeField(
help_text="Last time a client requested this virtual feed"
),
),
("created_at", models.DateTimeField(auto_now_add=True)),
(
"profile",
models.OneToOneField(
help_text="Profile that stores the bridged data",
on_delete=django.db.models.deletion.CASCADE,
related_name="rss_bridge",
to="feeds.profile",
),
),
],
options={
"ordering": ["source_url"],
},
),
]

0
app/bridge/migrations/__init__.py Normal file
View file

125
app/bridge/models.py Normal file
View file

@ -0,0 +1,125 @@
from urllib.parse import quote, urlparse
from django.conf import settings
from django.db import models
# Path fragments that identify a bridge virtual feed, on this relay or any other
BRIDGE_PATH_MARKERS = ("/bridge/activitypub/", "/bridge/rss/")
def is_bridge_feed_url(url):
"""
True when the URL points to a bridge virtual feed.
Bridges help users connect to external content but are not real
Org Social accounts, so they must stay out of the feed registry
and the statistics.
"""
path = urlparse(url).path
return any(marker in path for marker in BRIDGE_PATH_MARKERS)
def bridge_urls_q(field):
"""
Q object matching rows whose `field` URL belongs to a bridge virtual
feed. Use with exclude() to leave bridges out of listings and stats.
"""
q = models.Q()
for marker in BRIDGE_PATH_MARKERS:
q |= models.Q(**{f"{field}__contains": marker})
return q
def bridge_base_url():
"""
Base URL of this relay, used to build the public URL of virtual feeds.
"""
domain = settings.SITE_DOMAIN
scheme = "http" if domain.startswith(("localhost", "127.")) else "https"
return f"{scheme}://{domain}"
def activitypub_feed_url(handle):
"""Public URL of the virtual feed of a bridged ActivityPub account."""
return f"{bridge_base_url()}/bridge/activitypub/@{handle}/"
def rss_feed_url(source_url):
"""Public URL of the virtual feed of a bridged RSS feed."""
return f"{bridge_base_url()}/bridge/rss/?url={quote(source_url, safe='')}"
class BridgedActivityPubAccount(models.Model):
"""
An ActivityPub account (e.g. a Mastodon user) exposed as a virtual
Org Social feed at /bridge/activitypub/@{handle}/.
"""
handle = models.CharField(
max_length=255,
unique=True,
help_text="Normalized handle in user@instance form (lowercase)",
)
actor_url = models.URLField(
max_length=500, help_text="ActivityPub actor document URL"
)
outbox_url = models.URLField(
max_length=500, blank=True, help_text="ActivityPub outbox URL"
)
profile = models.OneToOneField(
"feeds.Profile",
on_delete=models.CASCADE,
related_name="activitypub_bridge",
help_text="Profile that stores the bridged data",
)
last_refreshed_at = models.DateTimeField(
null=True, blank=True, help_text="Last successful fetch from the origin"
)
last_accessed_at = models.DateTimeField(
help_text="Last time a client requested this virtual feed"
)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
ordering = ["handle"]
def __str__(self):
return f"@{self.handle}"
@property
def feed_url(self):
return activitypub_feed_url(self.handle)
class BridgedRssFeed(models.Model):
"""
An RSS/Atom feed exposed as a virtual Org Social feed at
/bridge/rss/?url={source_url}.
"""
source_url = models.URLField(
max_length=500, unique=True, help_text="URL of the RSS/Atom feed"
)
profile = models.OneToOneField(
"feeds.Profile",
on_delete=models.CASCADE,
related_name="rss_bridge",
help_text="Profile that stores the bridged data",
)
last_refreshed_at = models.DateTimeField(
null=True, blank=True, help_text="Last successful fetch from the origin"
)
last_accessed_at = models.DateTimeField(
help_text="Last time a client requested this virtual feed"
)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
ordering = ["source_url"]
def __str__(self):
return self.source_url
@property
def feed_url(self):
return rss_feed_url(self.source_url)

54
app/bridge/org_renderer.py Normal file
View file

@ -0,0 +1,54 @@
"""
Renders a bridged Profile and its Posts as a virtual social.org file.
The output follows the Org Social syntax so that any client can consume
the bridge URL with a plain #+FOLLOW: line.
"""
from .html_to_org import escape_org_lines
def _meta_value(value):
"""Metadata values must stay on a single line."""
return " ".join(str(value).split())
def render_profile_org(profile, posts):
"""
Build the virtual social.org content for a bridged profile.
Args:
profile: app.feeds.models.Profile instance
posts: iterable of app.feeds.models.Post, oldest first
Returns:
str: the Org Social file content
"""
lines = []
lines.append(f"#+TITLE: {_meta_value(profile.title) or profile.nick}")
lines.append(f"#+NICK: {_meta_value(profile.nick).replace(' ', '_')}")
if profile.description:
lines.append(f"#+DESCRIPTION: {_meta_value(profile.description)}")
if profile.avatar:
lines.append(f"#+AVATAR: {_meta_value(profile.avatar)}")
for link in profile.links.all():
lines.append(f"#+LINK: {_meta_value(link.url)}")
lines.append("")
lines.append("* Posts")
for post in posts:
lines.append(f"** {post.post_id}")
lines.append(":PROPERTIES:")
if post.language:
lines.append(f":LANG: {_meta_value(post.language)}")
if post.tags:
lines.append(f":TAGS: {_meta_value(post.tags)}")
lines.append(":END:")
lines.append("")
content = escape_org_lines(post.content.strip())
if content:
lines.append(content)
lines.append("")
return "\n".join(lines) + "\n"

139
app/bridge/rss.py Normal file
View file

@ -0,0 +1,139 @@
"""
RSS/Atom fetching for the bridge, normalized into the same structure
used by the ActivityPub bridge.
Each entry becomes a post whose body is the entry title as an Org
sub-heading (***), the converted entry content, and a link to the
original article.
"""
import calendar
import logging
import re
from datetime import datetime, timezone as dt_timezone
from urllib.parse import urlparse
import feedparser
from .fetching import BridgeError, safe_get
from .html_to_org import html_to_org
logger = logging.getLogger(__name__)
MAX_POSTS = 40
_NICK_ALLOWED_RE = re.compile(r"[^A-Za-z0-9_-]+")
def _derive_nick(title, source_url):
"""Build a spaces-free nick from the feed title or the feed host."""
candidate = _NICK_ALLOWED_RE.sub("_", (title or "").strip()).strip("_")
if candidate:
return candidate
host = urlparse(source_url).hostname or "feed"
return host.replace(".", "_")
def _entry_post_id(entry):
"""Org Social post ID (UTC RFC 3339) from the entry date, or None."""
parsed_time = entry.get("published_parsed") or entry.get("updated_parsed")
if not parsed_time:
return None
timestamp = calendar.timegm(parsed_time)
return datetime.fromtimestamp(timestamp, dt_timezone.utc).strftime(
"%Y-%m-%dT%H:%M:%S+00:00"
)
def _next_free_id(post_id, seen_ids):
"""
Entries sharing the same timestamp (e.g. date-only feeds) get IDs
shifted forward one second at a time so no post is lost.
"""
while post_id in seen_ids:
parsed = datetime.strptime(post_id, "%Y-%m-%dT%H:%M:%S+00:00")
shifted = parsed.timestamp() + 1
post_id = datetime.fromtimestamp(shifted, dt_timezone.utc).strftime(
"%Y-%m-%dT%H:%M:%S+00:00"
)
return post_id
def _entry_body(entry):
contents = entry.get("content") or []
if contents and contents[0].get("value"):
html = contents[0]["value"]
else:
html = entry.get("summary") or ""
return html_to_org(html)
def _entry_to_post(entry, seen_ids):
post_id = _entry_post_id(entry)
if post_id is None:
logger.warning(
f"Skipping RSS entry without date: {entry.get('link', '(no link)')}"
)
return None
post_id = _next_free_id(post_id, seen_ids)
parts = []
title = " ".join((entry.get("title") or "").split())
if title:
parts.append(f"*** {title}")
body = _entry_body(entry)
if body:
parts.append(body)
link = (entry.get("link") or "").strip()
if link and "]" not in link:
parts.append(f"[[{link}]]")
tags = " ".join(
tag.get("term", "").strip().replace(" ", "-")
for tag in entry.get("tags", [])
if tag.get("term")
)
return {
"id": post_id,
"content": "\n\n".join(parts),
"tags": tags,
"language": "",
}
def fetch_rss_feed(source_url):
"""
Fetch an RSS/Atom feed and normalize it for the bridge.
Returns a dict with metadata and posts.
Raises BridgeError when the feed cannot be fetched or parsed.
"""
raw = safe_get(source_url)
document = feedparser.parse(raw)
feed_info = document.get("feed", {})
if not document.get("entries") and not feed_info.get("title"):
raise BridgeError(f"Not a valid RSS/Atom feed: {source_url}")
title = " ".join((feed_info.get("title") or "").split())
image = feed_info.get("image") or {}
metadata = {
"title": title or source_url,
"nick": _derive_nick(title, source_url),
"description": " ".join((feed_info.get("subtitle") or "").split()),
"avatar": image.get("href", ""),
"link": feed_info.get("link") or source_url,
"language": (feed_info.get("language") or "").split("-")[0],
}
posts = []
seen_ids = set()
for entry in document.get("entries", [])[:MAX_POSTS]:
post = _entry_to_post(entry, seen_ids)
if post is None:
continue
seen_ids.add(post["id"])
posts.append(post)
return {"metadata": metadata, "posts": posts}

185
app/bridge/store.py Normal file
View file

@ -0,0 +1,185 @@
"""
Persistence and orchestration for bridged accounts.
Normalized bridge data (from activitypub.py or rss.py) is stored in the
same Profile/Post tables used for real Org Social feeds, so every
existing relay endpoint (/profile/, /search/, /rss.xml, ...) works on
bridged accounts transparently.
Bridged posts never queue Webmentions nor publish notifications: the
content belongs to external authors, the relay only mirrors it.
"""
import hashlib
import logging
from dateutil import parser as date_parser
from django.utils import timezone
from app.feeds.models import Post, Profile, ProfileLink
from . import activitypub, rss
from .models import (
BridgedActivityPubAccount,
BridgedRssFeed,
activitypub_feed_url,
rss_feed_url,
)
logger = logging.getLogger(__name__)
def store_bridge_data(feed_url, data):
"""
Create or update the Profile and Posts of a bridged account.
Args:
feed_url: public URL of the virtual feed on this relay
data: normalized dict with "metadata" and "posts"
Returns:
Profile: the stored profile
"""
metadata = data["metadata"]
posts_data = data["posts"]
content_hash = hashlib.md5(f"{metadata}{posts_data}".encode()).hexdigest()
nick = " ".join((metadata.get("nick") or "").split()).replace(" ", "_")
profile_fields = {
"title": metadata.get("title", ""),
"nick": nick,
"description": metadata.get("description", ""),
"avatar": metadata.get("avatar", ""),
"language": metadata.get("language", ""),
"version": content_hash,
}
profile, created = Profile.objects.get_or_create(
feed=feed_url, defaults=profile_fields
)
if not created:
if profile.version == content_hash:
return profile
for field, value in profile_fields.items():
setattr(profile, field, value)
profile.save()
profile.links.all().delete()
link = (metadata.get("link") or "").strip()
if link:
ProfileLink.objects.create(profile=profile, url=link)
fetched_ids = set()
oldest_created_at = None
for post_data in posts_data:
post_id = post_data["id"]
try:
post_created_at = date_parser.parse(post_id)
except (ValueError, OverflowError):
logger.warning(f"Skipping bridged post with invalid ID: {post_id}")
continue
fetched_ids.add(post_id)
if oldest_created_at is None or post_created_at < oldest_created_at:
oldest_created_at = post_created_at
post, post_created = Post.objects.get_or_create(
profile=profile,
post_id=post_id,
defaults={
"content": post_data["content"],
"tags": post_data.get("tags", "")[:500],
"language": post_data.get("language", "")[:10],
"created_at": post_created_at,
},
)
if not post_created:
post.content = post_data["content"]
post.tags = post_data.get("tags", "")[:500]
post.language = post_data.get("language", "")[:10]
post.save()
# Posts deleted at the origin disappear from the fetched window;
# remove them, but never touch posts older than what was fetched
if fetched_ids and oldest_created_at is not None:
Post.objects.filter(profile=profile, created_at__gte=oldest_created_at).exclude(
post_id__in=fetched_ids
).delete()
return profile
def create_activitypub_bridge(handle):
"""
Fetch an ActivityPub account for the first time and create its bridge.
Raises BridgeError when the account cannot be fetched.
"""
user, instance = handle.split("@", 1)
data = activitypub.fetch_account(user, instance)
profile = store_bridge_data(activitypub_feed_url(handle), data)
now = timezone.now()
bridge = BridgedActivityPubAccount.objects.create(
handle=handle,
actor_url=data["actor_url"],
outbox_url=data["outbox_url"],
profile=profile,
last_refreshed_at=now,
last_accessed_at=now,
)
logger.info(f"Created ActivityPub bridge for @{handle}")
return bridge
def refresh_activitypub_bridge(bridge):
"""
Refetch a bridged ActivityPub account and update its stored data.
Raises BridgeError when the origin cannot be fetched.
"""
user, instance = bridge.handle.split("@", 1)
data = activitypub.fetch_account(user, instance)
store_bridge_data(bridge.feed_url, data)
bridge.actor_url = data["actor_url"] or bridge.actor_url
bridge.outbox_url = data["outbox_url"] or bridge.outbox_url
bridge.last_refreshed_at = timezone.now()
bridge.save()
def create_rss_bridge(source_url):
"""
Fetch an RSS/Atom feed for the first time and create its bridge.
Raises BridgeError when the feed cannot be fetched.
"""
data = rss.fetch_rss_feed(source_url)
profile = store_bridge_data(rss_feed_url(source_url), data)
now = timezone.now()
bridge = BridgedRssFeed.objects.create(
source_url=source_url,
profile=profile,
last_refreshed_at=now,
last_accessed_at=now,
)
logger.info(f"Created RSS bridge for {source_url}")
return bridge
def refresh_rss_bridge(bridge):
"""
Refetch a bridged RSS feed and update its stored data.
Raises BridgeError when the origin cannot be fetched.
"""
data = rss.fetch_rss_feed(bridge.source_url)
store_bridge_data(bridge.feed_url, data)
bridge.last_refreshed_at = timezone.now()
bridge.save()

110
app/bridge/tasks.py Normal file
View file

@ -0,0 +1,110 @@
"""
Periodic maintenance of bridged accounts.
Only bridges requested recently are refreshed; dormant ones are
reactivated by the inline refresh on their next GET, and bridges nobody
has requested for a long time are deleted together with their profiles.
"""
import logging
from datetime import timedelta
from huey import crontab
from huey.contrib.djhuey import periodic_task
logger = logging.getLogger(__name__)
# Bridges accessed within this window are refreshed periodically
BRIDGE_ACTIVE_DAYS = 7
# Bridges not accessed for this long are deleted
BRIDGE_STALE_DAYS = 90
def _refresh_bridges_impl():
"""
Refresh every bridge accessed within the active window.
Returns:
dict: Counters of refreshed / failed bridges
"""
from django.utils import timezone
from .fetching import BridgeError
from .models import BridgedActivityPubAccount, BridgedRssFeed
from .store import refresh_activitypub_bridge, refresh_rss_bridge
cutoff = timezone.now() - timedelta(days=BRIDGE_ACTIVE_DAYS)
counters = {"refreshed": 0, "failed": 0}
refresh_plan = [
(BridgedActivityPubAccount, refresh_activitypub_bridge),
(BridgedRssFeed, refresh_rss_bridge),
]
for model, refresh in refresh_plan:
for bridge in model.objects.filter(last_accessed_at__gte=cutoff):
try:
refresh(bridge)
counters["refreshed"] += 1
except BridgeError as e:
counters["failed"] += 1
logger.warning(f"Failed to refresh bridge {bridge}: {e}")
except Exception as e:
counters["failed"] += 1
logger.error(f"Unexpected error refreshing bridge {bridge}: {e}")
logger.info(
f"Bridge refresh completed. "
f"Refreshed: {counters['refreshed']}, Failed: {counters['failed']}"
)
return counters
@periodic_task(crontab(minute="*/15")) # Run every 15 minutes
def refresh_bridges():
"""Periodic task to keep active bridged accounts up to date."""
import django
django.setup()
return _refresh_bridges_impl()
def _cleanup_stale_bridges_impl():
"""
Delete bridges (and their profiles with all posts) that nobody has
requested in BRIDGE_STALE_DAYS days.
Returns:
int: Number of bridges deleted
"""
from django.utils import timezone
from app.feeds.models import Profile
cutoff = timezone.now() - timedelta(days=BRIDGE_STALE_DAYS)
# Deleting the Profile cascades to the bridge row and its posts
stale_profiles = Profile.objects.filter(
activitypub_bridge__last_accessed_at__lt=cutoff
) | Profile.objects.filter(rss_bridge__last_accessed_at__lt=cutoff)
deleted = 0
for profile in stale_profiles:
logger.info(f"Deleting stale bridge profile: {profile.feed}")
profile.delete()
deleted += 1
if deleted:
logger.info(f"Stale bridge cleanup completed. Deleted {deleted} bridges")
return deleted
@periodic_task(crontab(day="*/3", hour=3, minute=0)) # Every 3 days at 3 AM
def cleanup_stale_bridges():
"""Periodic task to delete bridges nobody requests anymore."""
import django
django.setup()
return _cleanup_stale_bridges_impl()

295
app/bridge/test_activitypub.py Normal file
View file

@ -0,0 +1,295 @@
import json
from unittest.mock import patch
from django.test import SimpleTestCase
from app.bridge.activitypub import (
MAX_POSTS,
fetch_account,
normalize_handle,
)
from app.bridge.fetching import BridgeError
WEBFINGER_URL = (
"https://mastodon.example/.well-known/webfinger"
"?resource=acct%3Aalice%40mastodon.example"
)
ACTOR_URL = "https://mastodon.example/users/alice"
OUTBOX_URL = "https://mastodon.example/users/alice/outbox"
OUTBOX_PAGE_URL = "https://mastodon.example/users/alice/outbox?page=true"
def _note_item(published, content, **extra):
note = {"type": "Note", "published": published, "content": content}
note.update(extra)
return {"type": "Create", "object": note}
def _default_responses():
return {
WEBFINGER_URL: {
"links": [
{"rel": "http://webfinger.net/rel/profile-page"},
{
"rel": "self",
"type": "application/activity+json",
"href": ACTOR_URL,
},
]
},
ACTOR_URL: {
"preferredUsername": "alice",
"name": "Alice",
"summary": "<p>My <strong>bio</strong></p>",
"icon": {"url": "https://mastodon.example/avatar.png"},
"url": "https://mastodon.example/@alice",
"outbox": OUTBOX_URL,
},
OUTBOX_URL: {"first": OUTBOX_PAGE_URL},
OUTBOX_PAGE_URL: {
"orderedItems": [
_note_item(
"2025-05-02T10:00:00Z",
"<p>Second post</p>",
tag=[{"type": "Hashtag", "name": "#emacs"}],
contentMap={"es": "<p>Segundo post</p>"},
),
_note_item(
"2025-05-01T18:00:00Z",
"<p>Reply post</p>",
inReplyTo="https://other.example/notes/1",
),
{"type": "Announce", "object": "https://other.example/notes/2"},
_note_item(
"2025-05-01T10:00:00Z",
"<p>First post</p>",
summary="Spoilers inside",
attachment=[
{
"type": "Document",
"url": "https://mastodon.example/media/cat.png",
}
],
),
]
},
}
def _fake_safe_get(responses):
def fake(url, accept=None):
if url not in responses:
raise BridgeError(f"Unexpected URL in test: {url}")
return json.dumps(responses[url]).encode()
return fake
class NormalizeHandleTest(SimpleTestCase):
"""Test cases for handle normalization."""
def test_valid_handle_is_lowercased_and_stripped(self):
# Given: A handle with leading @, capitals and spaces around
raw = " @Alice@Mastodon.Example "
# When: It is normalized
result = normalize_handle(raw)
# Then: The canonical lowercase form is returned
self.assertEqual(result, "alice@mastodon.example")
def test_handle_without_at_prefix_is_accepted(self):
# Given: A handle without the leading @
# When: It is normalized
# Then: It is valid
self.assertEqual(
normalize_handle("alice@mastodon.example"), "alice@mastodon.example"
)
def test_invalid_handles_return_none(self):
# Given: Handles missing parts or with invalid characters
invalid = [
"alice",
"alice@",
"@instance-only",
"alice@nodots",
"ali ce@mastodon.example",
"alice@mastodon.example/evil",
"",
None,
]
# When/Then: None of them normalize
for handle in invalid:
self.assertIsNone(normalize_handle(handle), handle)
class FetchAccountTest(SimpleTestCase):
"""Test cases for the ActivityPub account fetcher."""
def test_fetches_profile_metadata_from_actor(self):
# Given: A reachable account with actor metadata
responses = _default_responses()
# When: The account is fetched
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
result = fetch_account("alice", "mastodon.example")
# Then: Metadata comes from the actor document
self.assertEqual(result["actor_url"], ACTOR_URL)
self.assertEqual(result["outbox_url"], OUTBOX_URL)
self.assertEqual(result["metadata"]["title"], "Alice")
self.assertEqual(result["metadata"]["nick"], "alice")
self.assertEqual(result["metadata"]["description"], "My *bio*")
self.assertEqual(
result["metadata"]["avatar"], "https://mastodon.example/avatar.png"
)
self.assertEqual(result["metadata"]["link"], "https://mastodon.example/@alice")
def test_only_top_level_notes_become_posts(self):
# Given: An outbox with a note, a reply and an announce
responses = _default_responses()
# When: The account is fetched
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
result = fetch_account("alice", "mastodon.example")
# Then: Only the two top-level notes are bridged
self.assertEqual(len(result["posts"]), 2)
ids = [post["id"] for post in result["posts"]]
self.assertEqual(
ids, ["2025-05-02T10:00:00+00:00", "2025-05-01T10:00:00+00:00"]
)
def test_post_carries_tags_language_cw_and_attachments(self):
# Given: Notes with hashtag, contentMap, summary and attachment
responses = _default_responses()
# When: The account is fetched
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
result = fetch_account("alice", "mastodon.example")
# Then: The second post has tags and language
tagged = result["posts"][0]
self.assertEqual(tagged["tags"], "emacs")
self.assertEqual(tagged["language"], "es")
# And: The first post has the CW line and the attachment link
with_cw = result["posts"][1]
self.assertTrue(with_cw["content"].startswith("CW: Spoilers inside"))
self.assertIn("[[https://mastodon.example/media/cat.png]]", with_cw["content"])
def test_notes_without_valid_date_are_skipped(self):
# Given: An outbox with a dateless note and a naive-date note
responses = _default_responses()
responses[OUTBOX_PAGE_URL] = {
"orderedItems": [
{"type": "Create", "object": {"type": "Note", "content": "<p>x</p>"}},
_note_item("2025-05-01T10:00:00", "<p>naive date</p>"),
_note_item("2025-05-01T10:00:00Z", "<p>good</p>"),
]
}
# When: The account is fetched
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
result = fetch_account("alice", "mastodon.example")
# Then: Only the note with a timezone-aware date survives
self.assertEqual(len(result["posts"]), 1)
self.assertEqual(result["posts"][0]["content"], "good")
def test_outbox_pagination_is_followed_up_to_max_posts(self):
# Given: An outbox split into two pages with many notes
page_two_url = "https://mastodon.example/users/alice/outbox?page=2"
responses = _default_responses()
responses[OUTBOX_PAGE_URL] = {
"orderedItems": [
_note_item(f"2025-05-02T10:00:{second:02d}Z", "<p>a</p>")
for second in range(30)
],
"next": page_two_url,
}
responses[page_two_url] = {
"orderedItems": [
_note_item(f"2025-05-01T10:00:{second:02d}Z", "<p>b</p>")
for second in range(30)
]
}
# When: The account is fetched
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
result = fetch_account("alice", "mastodon.example")
# Then: Both pages are read but capped at MAX_POSTS
self.assertEqual(len(result["posts"]), MAX_POSTS)
def test_webfinger_without_actor_link_raises(self):
# Given: A WebFinger response without a self link
responses = _default_responses()
responses[WEBFINGER_URL] = {"links": [{"rel": "other"}]}
# When/Then: Fetching raises a BridgeError
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
with self.assertRaises(BridgeError):
fetch_account("alice", "mastodon.example")
def test_invalid_json_raises_bridge_error(self):
# Given: A server answering HTML instead of JSON
def fake(url, accept=None):
return b"<html>not json</html>"
# When/Then: Fetching raises a BridgeError
with patch("app.bridge.activitypub.safe_get", side_effect=fake):
with self.assertRaises(BridgeError):
fetch_account("alice", "mastodon.example")
def test_actor_without_outbox_returns_profile_without_posts(self):
# Given: An actor document without an outbox
responses = _default_responses()
del responses[ACTOR_URL]["outbox"]
# When: The account is fetched
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
result = fetch_account("alice", "mastodon.example")
# Then: The profile exists with no posts
self.assertEqual(result["posts"], [])
self.assertEqual(result["outbox_url"], "")
def test_outbox_with_inline_items_needs_no_extra_page(self):
# Given: An outbox embedding orderedItems directly
responses = _default_responses()
responses[OUTBOX_URL] = {
"orderedItems": [_note_item("2025-05-01T10:00:00Z", "<p>inline</p>")]
}
# When: The account is fetched
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
result = fetch_account("alice", "mastodon.example")
# Then: The inline note is bridged
self.assertEqual(len(result["posts"]), 1)
self.assertEqual(result["posts"][0]["content"], "inline")

243
app/bridge/test_html_to_org.py Normal file
View file

@ -0,0 +1,243 @@
from django.test import SimpleTestCase
from app.bridge.html_to_org import escape_org_lines, html_to_org, html_to_text
class HtmlToOrgTest(SimpleTestCase):
"""Test cases for the HTML to Org converter."""
def test_paragraphs_become_blank_line_separated(self):
# Given: HTML with two paragraphs
html = "<p>First paragraph</p><p>Second paragraph</p>"
# When: It is converted to Org
result = html_to_org(html)
# Then: Paragraphs are separated by a blank line
self.assertEqual(result, "First paragraph\n\nSecond paragraph")
def test_br_becomes_newline(self):
# Given: HTML with a line break inside a paragraph
html = "<p>First line<br>Second line</p>"
# When: It is converted to Org
result = html_to_org(html)
# Then: The break is kept as a single newline
self.assertEqual(result, "First line\nSecond line")
def test_link_with_text_becomes_org_link(self):
# Given: An anchor with its own text
html = '<p>Read <a href="https://example.com/post">this article</a></p>'
# When: It is converted to Org
result = html_to_org(html)
# Then: An Org link with description is produced
self.assertEqual(result, "Read [[https://example.com/post][this article]]")
def test_link_with_url_as_text_becomes_plain_org_link(self):
# Given: An anchor whose text is the URL itself
html = '<a href="https://example.com">https://example.com</a>'
# When: It is converted to Org
result = html_to_org(html)
# Then: A plain Org link is produced
self.assertEqual(result, "[[https://example.com]]")
def test_hashtag_and_mention_links_stay_as_plain_text(self):
# Given: Mastodon-style hashtag and mention anchors
html = (
'<p>Hi <a href="https://m.example/@bob" class="u-url mention">'
"@bob</a> about "
'<a href="https://m.example/tags/emacs" class="mention hashtag">'
"#emacs</a></p>"
)
# When: It is converted to Org
result = html_to_org(html)
# Then: No Org links are produced, only their text
self.assertEqual(result, "Hi @bob about #emacs")
def test_emphasis_tags_become_org_markers(self):
# Given: HTML with bold, italic and code
html = "<p><strong>bold</strong> <em>italic</em> <code>code</code></p>"
# When: It is converted to Org
result = html_to_org(html)
# Then: Org emphasis markers are used
self.assertEqual(result, "*bold* /italic/ ~code~")
def test_list_items_become_org_list(self):
# Given: An unordered list
html = "<ul><li>One</li><li>Two</li></ul>"
# When: It is converted to Org
result = html_to_org(html)
# Then: Each item becomes a dash line
self.assertEqual(result, "- One\n- Two")
def test_blockquote_becomes_quote_block(self):
# Given: A quoted paragraph
html = "<blockquote><p>Wise words</p></blockquote>"
# When: It is converted to Org
result = html_to_org(html)
# Then: The text is wrapped in a quote block
self.assertEqual(result, "#+BEGIN_QUOTE\n\nWise words\n\n#+END_QUOTE")
def test_pre_becomes_example_block(self):
# Given: Preformatted content
html = "<pre>print('hi')</pre>"
# When: It is converted to Org
result = html_to_org(html)
# Then: The content is wrapped in an example block preserving text
self.assertEqual(result, "#+BEGIN_EXAMPLE\nprint('hi')\n#+END_EXAMPLE")
def test_invisible_spans_are_skipped(self):
# Given: A Mastodon-style shortened URL anchor
html = (
'<a href="https://example.com/very/long/path">'
'<span class="invisible">https://</span>'
'<span class="ellipsis">example.com/very</span></a>'
)
# When: It is converted to Org
result = html_to_org(html)
# Then: The invisible prefix is not part of the link text
self.assertEqual(
result, "[[https://example.com/very/long/path][example.com/very]]"
)
def test_image_becomes_org_link_with_alt_text(self):
# Given: An image with alt text and another without it
html = (
'<p><img src="https://example.com/a.png" alt="A comic"/></p>'
'<p><img src="https://example.com/b.png"/></p>'
)
# When: It is converted to Org
result = html_to_org(html)
# Then: Images become Org links, using alt as description
self.assertEqual(
result,
"[[https://example.com/a.png][A comic]]\n\n[[https://example.com/b.png]]",
)
def test_image_wrapped_in_anchor_keeps_only_the_image_link(self):
# Given: An image wrapped in an anchor (common in RSS feeds)
html = (
'<a href="https://example.com/post">'
'<img src="https://example.com/a.png" alt="A comic"/></a>'
)
# When: It is converted to Org
result = html_to_org(html)
# Then: No nested Org links are produced
self.assertEqual(result, "[[https://example.com/a.png][A comic]]")
def test_html_entities_are_decoded(self):
# Given: HTML with entities
html = "<p>Fish &amp; chips &lt;3</p>"
# When: It is converted to Org
result = html_to_org(html)
# Then: Entities are decoded
self.assertEqual(result, "Fish & chips <3")
def test_headline_injection_is_escaped(self):
# Given: Remote content that looks like Org Social headlines
html = "<p>** 2025-01-01T00:00:00+00:00<br>* Posts</p>"
# When: It is converted to Org
result = html_to_org(html)
# Then: The lines are prefixed so they cannot break the feed
for line in result.split("\n"):
self.assertFalse(line.startswith("*"))
def test_empty_and_missing_html_return_empty_string(self):
# Given: Empty and missing input
# When: They are converted
# Then: The result is an empty string
self.assertEqual(html_to_org(""), "")
self.assertEqual(html_to_org(None), "")
def test_unknown_tags_are_stripped_keeping_text(self):
# Given: HTML with tags the converter does not know
html = "<article><section>Some text</section></article>"
# When: It is converted to Org
result = html_to_org(html)
# Then: The text survives without the tags
self.assertEqual(result, "Some text")
class HtmlToTextTest(SimpleTestCase):
"""Test cases for the single-line text converter."""
def test_multiline_html_becomes_single_line(self):
# Given: HTML with several paragraphs
html = "<p>Bio line one</p><p>Bio line two</p>"
# When: It is converted to plain text
result = html_to_text(html)
# Then: Everything is collapsed into one line
self.assertEqual(result, "Bio line one Bio line two")
class EscapeOrgLinesTest(SimpleTestCase):
"""Test cases for headline escaping."""
def test_one_and_two_asterisk_headlines_are_escaped(self):
# Given: Text with dangerous headline-like lines
text = "* Posts\n** 2025-01-01T00:00:00+00:00\nnormal"
# When: It is escaped
result = escape_org_lines(text)
# Then: Dangerous lines are prefixed with a space
self.assertEqual(result, " * Posts\n ** 2025-01-01T00:00:00+00:00\nnormal")
def test_post_titles_with_three_asterisks_are_kept(self):
# Given: A legal Org Social post title line
text = "*** My title"
# When: It is escaped
result = escape_org_lines(text)
# Then: The title line is untouched
self.assertEqual(result, "*** My title")
def test_bold_text_at_line_start_is_kept(self):
# Given: A line starting with Org bold (no space after asterisk)
text = "*bold* rest"
# When: It is escaped
result = escape_org_lines(text)
# Then: The line is untouched
self.assertEqual(result, "*bold* rest")
def test_escaping_is_idempotent(self):
# Given: Already escaped text
text = escape_org_lines("** 2025-01-01T00:00:00+00:00")
# When: It is escaped again
result = escape_org_lines(text)
# Then: Nothing changes
self.assertEqual(result, text)

42
app/bridge/test_models.py Normal file
View file

@ -0,0 +1,42 @@
from django.test import SimpleTestCase
from app.bridge.models import is_bridge_feed_url
class IsBridgeFeedUrlTest(SimpleTestCase):
"""Test cases for is_bridge_feed_url using Given/When/Then structure."""
def test_detects_rss_bridge_url(self):
"""Test an RSS bridge URL is detected as a bridge feed."""
# Given: The URL of an RSS bridge on any relay
url = (
"https://relay.org-social.org/bridge/rss/"
"?url=https%3A%2F%2Frss.arxiv.org%2Frss%2Fquant-ph"
)
# When / Then: It is identified as a bridge feed
self.assertTrue(is_bridge_feed_url(url))
def test_detects_activitypub_bridge_url(self):
"""Test an ActivityPub bridge URL is detected as a bridge feed."""
# Given: The URL of an ActivityPub bridge on any relay
url = "https://other-relay.org/bridge/activitypub/@user@instance.tld/"
# When / Then: It is identified as a bridge feed
self.assertTrue(is_bridge_feed_url(url))
def test_regular_feed_is_not_a_bridge(self):
"""Test a regular social.org feed is not detected as a bridge."""
# Given: The URL of a real Org Social feed
url = "https://example.com/social.org"
# When / Then: It is not identified as a bridge feed
self.assertFalse(is_bridge_feed_url(url))
def test_bridge_marker_in_query_string_is_ignored(self):
"""Test the bridge marker only matches in the URL path."""
# Given: A real feed whose query string mentions a bridge path
url = "https://example.com/social.org?note=/bridge/rss/"
# When / Then: It is not identified as a bridge feed
self.assertFalse(is_bridge_feed_url(url))

118
app/bridge/test_org_renderer.py Normal file
View file

@ -0,0 +1,118 @@
from django.test import TestCase
from app.bridge.org_renderer import render_profile_org
from app.feeds.models import Post, Profile, ProfileLink
class RenderProfileOrgTest(TestCase):
"""Test cases for the virtual social.org renderer."""
def setUp(self):
# Given: A bridged profile with a link and two posts
self.profile = Profile.objects.create(
feed="http://localhost:8080/bridge/activitypub/@alice@m.example/",
title="Alice in Wonderland",
nick="alice",
description="Just a test account",
avatar="https://m.example/avatar.png",
version="v1",
)
ProfileLink.objects.create(profile=self.profile, url="https://m.example/@alice")
Post.objects.create(
profile=self.profile,
post_id="2025-05-01T10:00:00+00:00",
content="Hello world",
created_at="2025-05-01T10:00:00+00:00",
)
Post.objects.create(
profile=self.profile,
post_id="2025-05-02T10:00:00+00:00",
content="Second post",
tags="emacs org",
language="en",
created_at="2025-05-02T10:00:00+00:00",
)
def test_renders_global_metadata(self):
# When: The profile is rendered
result = render_profile_org(self.profile, self.profile.posts.all())
# Then: All global metadata lines are present
self.assertIn("#+TITLE: Alice in Wonderland\n", result)
self.assertIn("#+NICK: alice\n", result)
self.assertIn("#+DESCRIPTION: Just a test account\n", result)
self.assertIn("#+AVATAR: https://m.example/avatar.png\n", result)
self.assertIn("#+LINK: https://m.example/@alice\n", result)
def test_renders_posts_section_with_properties(self):
# When: The profile is rendered oldest post first
posts = self.profile.posts.order_by("created_at")
result = render_profile_org(self.profile, posts)
# Then: Posts appear under * Posts with their properties
self.assertIn("* Posts\n", result)
self.assertIn("** 2025-05-01T10:00:00+00:00\n", result)
self.assertIn("** 2025-05-02T10:00:00+00:00\n", result)
self.assertIn(":TAGS: emacs org\n", result)
self.assertIn(":LANG: en\n", result)
self.assertIn("Hello world\n", result)
# And: The first post comes before the second
self.assertLess(
result.index("2025-05-01T10:00:00+00:00"),
result.index("2025-05-02T10:00:00+00:00"),
)
def test_nick_with_spaces_is_sanitized(self):
# Given: A profile whose nick contains spaces
self.profile.nick = "alice cooper"
# When: The profile is rendered
result = render_profile_org(self.profile, [])
# Then: The nick has no spaces
self.assertIn("#+NICK: alice_cooper\n", result)
def test_metadata_never_spans_multiple_lines(self):
# Given: A profile description containing newlines
self.profile.description = "line one\nline two"
# When: The profile is rendered
result = render_profile_org(self.profile, [])
# Then: The description is collapsed into a single line
self.assertIn("#+DESCRIPTION: line one line two\n", result)
def test_post_content_headlines_are_escaped(self):
# Given: A post whose content looks like an Org Social headline
Post.objects.create(
profile=self.profile,
post_id="2025-05-03T10:00:00+00:00",
content="** 2030-01-01T00:00:00+00:00\ninjected",
created_at="2025-05-03T10:00:00+00:00",
)
# When: The profile is rendered
result = render_profile_org(
self.profile, self.profile.posts.order_by("created_at")
)
# Then: The injected headline is escaped
self.assertIn(" ** 2030-01-01T00:00:00+00:00\n", result)
self.assertNotIn("\n** 2030-01-01T00:00:00+00:00\n", result)
def test_empty_optional_metadata_is_omitted(self):
# Given: A profile with no description, avatar nor links
profile = Profile.objects.create(
feed="http://localhost:8080/bridge/rss/?url=x",
title="Bare",
nick="bare",
version="v1",
)
# When: The profile is rendered
result = render_profile_org(profile, [])
# Then: Optional lines are not present
self.assertNotIn("#+DESCRIPTION:", result)
self.assertNotIn("#+AVATAR:", result)
self.assertNotIn("#+LINK:", result)

172
app/bridge/test_rss.py Normal file
View file

@ -0,0 +1,172 @@
from unittest.mock import patch
from django.test import SimpleTestCase
from app.bridge.fetching import BridgeError
from app.bridge.rss import fetch_rss_feed
RSS_SAMPLE = b"""<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
<channel>
<title>My Blog</title>
<link>https://blog.example</link>
<description>Notes about software</description>
<language>en-us</language>
<image>
<url>https://blog.example/logo.png</url>
<title>My Blog</title>
<link>https://blog.example</link>
</image>
<item>
<title>Second article</title>
<link>https://blog.example/second</link>
<pubDate>Fri, 02 May 2025 10:00:00 GMT</pubDate>
<category>emacs</category>
<category>org mode</category>
<description>&lt;p&gt;Second &lt;strong&gt;body&lt;/strong&gt;&lt;/p&gt;</description>
</item>
<item>
<title>First article</title>
<link>https://blog.example/first</link>
<pubDate>Thu, 01 May 2025 10:00:00 GMT</pubDate>
<description>First body</description>
</item>
<item>
<title>No date article</title>
<link>https://blog.example/undated</link>
<description>Should be skipped</description>
</item>
</channel>
</rss>
"""
ATOM_SAMPLE = b"""<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>Atom Feed</title>
<subtitle>An atom feed</subtitle>
<link href="https://atom.example"/>
<entry>
<title>Atom entry</title>
<link href="https://atom.example/entry"/>
<updated>2025-05-01T10:00:00Z</updated>
<content type="html">&lt;p&gt;Atom body&lt;/p&gt;</content>
</entry>
</feed>
"""
DUPLICATED_DATES_SAMPLE = b"""<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
<channel>
<title>Daily</title>
<link>https://daily.example</link>
<item>
<title>Post A</title>
<link>https://daily.example/a</link>
<pubDate>Thu, 01 May 2025 00:00:00 GMT</pubDate>
</item>
<item>
<title>Post B</title>
<link>https://daily.example/b</link>
<pubDate>Thu, 01 May 2025 00:00:00 GMT</pubDate>
</item>
</channel>
</rss>
"""
class FetchRssFeedTest(SimpleTestCase):
"""Test cases for the RSS/Atom fetcher."""
def test_rss2_metadata_is_extracted(self):
# Given: A standard RSS 2.0 feed
with patch("app.bridge.rss.safe_get", return_value=RSS_SAMPLE):
# When: The feed is fetched
result = fetch_rss_feed("https://blog.example/feed.xml")
# Then: The channel metadata maps to profile metadata
metadata = result["metadata"]
self.assertEqual(metadata["title"], "My Blog")
self.assertEqual(metadata["nick"], "My_Blog")
self.assertEqual(metadata["description"], "Notes about software")
self.assertEqual(metadata["avatar"], "https://blog.example/logo.png")
self.assertEqual(metadata["link"], "https://blog.example")
self.assertEqual(metadata["language"], "en")
def test_entries_become_posts_with_title_body_and_link(self):
# Given: A standard RSS 2.0 feed
with patch("app.bridge.rss.safe_get", return_value=RSS_SAMPLE):
# When: The feed is fetched
result = fetch_rss_feed("https://blog.example/feed.xml")
# Then: Dated entries become posts with UTC IDs
ids = [post["id"] for post in result["posts"]]
self.assertEqual(
ids, ["2025-05-02T10:00:00+00:00", "2025-05-01T10:00:00+00:00"]
)
# And: The body has the title heading, converted content and link
second = result["posts"][0]
self.assertIn("*** Second article", second["content"])
self.assertIn("Second *body*", second["content"])
self.assertIn("[[https://blog.example/second]]", second["content"])
# And: Categories become tags with spaces replaced
self.assertEqual(second["tags"], "emacs org-mode")
def test_entries_without_date_are_skipped(self):
# Given: A feed with a dateless entry
with patch("app.bridge.rss.safe_get", return_value=RSS_SAMPLE):
# When: The feed is fetched
result = fetch_rss_feed("https://blog.example/feed.xml")
# Then: The dateless entry is not bridged
contents = " ".join(post["content"] for post in result["posts"])
self.assertNotIn("No date article", contents)
def test_atom_feeds_are_supported(self):
# Given: An Atom feed
with patch("app.bridge.rss.safe_get", return_value=ATOM_SAMPLE):
# When: The feed is fetched
result = fetch_rss_feed("https://atom.example/feed.atom")
# Then: Metadata and the entry are bridged
self.assertEqual(result["metadata"]["title"], "Atom Feed")
self.assertEqual(len(result["posts"]), 1)
self.assertEqual(result["posts"][0]["id"], "2025-05-01T10:00:00+00:00")
self.assertIn("Atom body", result["posts"][0]["content"])
def test_entries_with_same_date_get_shifted_ids(self):
# Given: A feed where two entries share the exact same date
with patch("app.bridge.rss.safe_get", return_value=DUPLICATED_DATES_SAMPLE):
# When: The feed is fetched
result = fetch_rss_feed("https://daily.example/feed.xml")
# Then: Both entries survive with distinct consecutive IDs
ids = sorted(post["id"] for post in result["posts"])
self.assertEqual(
ids, ["2025-05-01T00:00:00+00:00", "2025-05-01T00:00:01+00:00"]
)
def test_non_feed_content_raises_bridge_error(self):
# Given: A URL answering plain HTML
with patch(
"app.bridge.rss.safe_get", return_value=b"<html><body>hi</body></html>"
):
# When/Then: Fetching raises a BridgeError
with self.assertRaises(BridgeError):
fetch_rss_feed("https://blog.example/not-a-feed")
def test_feed_without_title_uses_host_as_nick(self):
# Given: A minimal feed without a channel title
sample = (
b'<?xml version="1.0"?><rss version="2.0"><channel>'
b"<item><title>x</title>"
b"<pubDate>Thu, 01 May 2025 00:00:00 GMT</pubDate></item>"
b"</channel></rss>"
)
with patch("app.bridge.rss.safe_get", return_value=sample):
# When: The feed is fetched
result = fetch_rss_feed("https://blog.example/feed.xml")
# Then: The nick is derived from the host
self.assertEqual(result["metadata"]["nick"], "blog_example")
# And: The title falls back to the source URL
self.assertEqual(result["metadata"]["title"], "https://blog.example/feed.xml")

166
app/bridge/test_store.py Normal file
View file

@ -0,0 +1,166 @@
from django.test import TestCase
from app.bridge.store import store_bridge_data
from app.feeds.models import Post, Profile
FEED_URL = "http://localhost:8080/bridge/activitypub/@alice@m.example/"
def _data(posts=None, nick="alice", title="Alice"):
return {
"metadata": {
"title": title,
"nick": nick,
"description": "Bio",
"avatar": "https://m.example/avatar.png",
"link": "https://m.example/@alice",
},
"posts": posts if posts is not None else [],
}
def _post(post_id, content, tags=""):
return {"id": post_id, "content": content, "tags": tags, "language": ""}
class StoreBridgeDataTest(TestCase):
"""Test cases for storing normalized bridge data."""
def test_creates_profile_with_posts_and_link(self):
# Given: Normalized data with two posts
data = _data(
posts=[
_post("2025-05-01T10:00:00+00:00", "First"),
_post("2025-05-02T10:00:00+00:00", "Second", tags="emacs"),
]
)
# When: The data is stored
profile = store_bridge_data(FEED_URL, data)
# Then: Profile, posts and link exist in the database
self.assertEqual(profile.feed, FEED_URL)
self.assertEqual(profile.nick, "alice")
self.assertEqual(profile.posts.count(), 2)
self.assertEqual(
list(profile.links.values_list("url", flat=True)),
["https://m.example/@alice"],
)
post = profile.posts.get(post_id="2025-05-02T10:00:00+00:00")
self.assertEqual(post.tags, "emacs")
def test_nick_with_spaces_is_sanitized(self):
# Given: Metadata with a nick containing spaces
data = _data(nick="alice cooper")
# When: The data is stored
profile = store_bridge_data(FEED_URL, data)
# Then: The nick has no spaces
self.assertEqual(profile.nick, "alice_cooper")
def test_unchanged_data_is_not_rewritten(self):
# Given: Data already stored once
data = _data(posts=[_post("2025-05-01T10:00:00+00:00", "First")])
profile = store_bridge_data(FEED_URL, data)
version = profile.version
# When: The exact same data is stored again
profile_again = store_bridge_data(FEED_URL, data)
# Then: The version hash does not change and posts are intact
self.assertEqual(profile_again.version, version)
self.assertEqual(profile_again.posts.count(), 1)
def test_changed_post_content_is_updated(self):
# Given: A stored post whose content changes at the origin
post_id = "2025-05-01T10:00:00+00:00"
store_bridge_data(FEED_URL, _data(posts=[_post(post_id, "Original")]))
# When: The new version is stored
store_bridge_data(FEED_URL, _data(posts=[_post(post_id, "Edited")]))
# Then: The post content is updated in place
profile = Profile.objects.get(feed=FEED_URL)
self.assertEqual(profile.posts.get(post_id=post_id).content, "Edited")
self.assertEqual(profile.posts.count(), 1)
def test_posts_deleted_at_origin_are_removed_within_window(self):
# Given: Three stored posts
store_bridge_data(
FEED_URL,
_data(
posts=[
_post("2025-05-01T10:00:00+00:00", "A"),
_post("2025-05-02T10:00:00+00:00", "B"),
_post("2025-05-03T10:00:00+00:00", "C"),
]
),
)
# When: The origin no longer returns the middle post
store_bridge_data(
FEED_URL,
_data(
posts=[
_post("2025-05-01T10:00:00+00:00", "A"),
_post("2025-05-03T10:00:00+00:00", "C"),
]
),
)
# Then: The deleted post is removed
profile = Profile.objects.get(feed=FEED_URL)
ids = set(profile.posts.values_list("post_id", flat=True))
self.assertEqual(
ids, {"2025-05-01T10:00:00+00:00", "2025-05-03T10:00:00+00:00"}
)
def test_posts_older_than_fetched_window_are_preserved(self):
# Given: An old post stored from a previous, deeper fetch
store_bridge_data(
FEED_URL, _data(posts=[_post("2025-01-01T10:00:00+00:00", "Old")])
)
# When: A later fetch only returns newer posts
store_bridge_data(
FEED_URL, _data(posts=[_post("2025-05-01T10:00:00+00:00", "New")])
)
# Then: The old post outside the fetched window survives
profile = Profile.objects.get(feed=FEED_URL)
ids = set(profile.posts.values_list("post_id", flat=True))
self.assertEqual(
ids, {"2025-01-01T10:00:00+00:00", "2025-05-01T10:00:00+00:00"}
)
def test_no_webmentions_are_queued_for_bridged_posts(self):
# Given: A bridged post containing an external link
from app.feeds.models import OutgoingWebmention
data = _data(
posts=[
_post(
"2025-05-01T10:00:00+00:00",
"Look at [[https://external.example/page][this]]",
)
]
)
# When: The data is stored
store_bridge_data(FEED_URL, data)
# Then: No outgoing webmention is created
self.assertEqual(OutgoingWebmention.objects.count(), 0)
def test_post_created_at_comes_from_post_id(self):
# Given: A post with a known timestamp ID
store_bridge_data(
FEED_URL, _data(posts=[_post("2025-05-01T10:00:00+00:00", "A")])
)
# When: The post is read back
post = Post.objects.get(post_id="2025-05-01T10:00:00+00:00")
# Then: created_at matches the ID timestamp
self.assertEqual(post.created_at.isoformat(), "2025-05-01T10:00:00+00:00")

160
app/bridge/test_tasks.py Normal file
View file

@ -0,0 +1,160 @@
from datetime import timedelta
from unittest.mock import patch
from django.test import TestCase
from django.utils import timezone
from app.bridge.fetching import BridgeError
from app.bridge.models import (
BridgedActivityPubAccount,
BridgedRssFeed,
activitypub_feed_url,
rss_feed_url,
)
from app.bridge.tasks import (
BRIDGE_ACTIVE_DAYS,
BRIDGE_STALE_DAYS,
_cleanup_stale_bridges_impl,
_refresh_bridges_impl,
)
from app.feeds.models import Post, Profile
def _make_ap_bridge(handle, accessed_days_ago=0):
profile = Profile.objects.create(
feed=activitypub_feed_url(handle),
title=handle,
nick=handle.split("@")[0],
version="v1",
)
return BridgedActivityPubAccount.objects.create(
handle=handle,
actor_url=f"https://{handle.split('@')[1]}/users/{handle.split('@')[0]}",
profile=profile,
last_accessed_at=timezone.now() - timedelta(days=accessed_days_ago),
)
def _make_rss_bridge(source_url, accessed_days_ago=0):
profile = Profile.objects.create(
feed=rss_feed_url(source_url),
title="Blog",
nick="blog",
version="v1",
)
return BridgedRssFeed.objects.create(
source_url=source_url,
profile=profile,
last_accessed_at=timezone.now() - timedelta(days=accessed_days_ago),
)
class RefreshBridgesTest(TestCase):
"""Test cases for the periodic bridge refresh."""
def test_only_recently_accessed_bridges_are_refreshed(self):
# Given: An active bridge and a dormant one
active = _make_ap_bridge("alice@m.example", accessed_days_ago=0)
_make_ap_bridge("bob@m.example", accessed_days_ago=BRIDGE_ACTIVE_DAYS + 1)
# When: The refresh task runs
with (
patch("app.bridge.store.refresh_activitypub_bridge") as mock_ap,
patch("app.bridge.store.refresh_rss_bridge"),
):
counters = _refresh_bridges_impl()
# Then: Only the active bridge is refreshed
self.assertEqual(counters, {"refreshed": 1, "failed": 0})
mock_ap.assert_called_once()
self.assertEqual(mock_ap.call_args[0][0].pk, active.pk)
def test_both_bridge_types_are_refreshed(self):
# Given: One active bridge of each type
_make_ap_bridge("alice@m.example")
_make_rss_bridge("https://blog.example/feed.xml")
# When: The refresh task runs
with (
patch("app.bridge.store.refresh_activitypub_bridge") as mock_ap,
patch("app.bridge.store.refresh_rss_bridge") as mock_rss,
):
counters = _refresh_bridges_impl()
# Then: Both are refreshed
self.assertEqual(counters, {"refreshed": 2, "failed": 0})
mock_ap.assert_called_once()
mock_rss.assert_called_once()
def test_failed_refreshes_are_counted_and_do_not_stop_the_task(self):
# Given: Two active bridges, the first origin failing
_make_ap_bridge("alice@m.example")
_make_ap_bridge("bob@m.example")
# When: The refresh task runs
with (
patch(
"app.bridge.store.refresh_activitypub_bridge",
side_effect=[BridgeError("down"), None],
),
patch("app.bridge.store.refresh_rss_bridge"),
):
counters = _refresh_bridges_impl()
# Then: The failure is counted and the other bridge still refreshed
self.assertEqual(counters, {"refreshed": 1, "failed": 1})
class CleanupStaleBridgesTest(TestCase):
"""Test cases for the stale bridge cleanup."""
def test_stale_bridges_are_deleted_with_profile_and_posts(self):
# Given: A stale bridge with posts and an active one
stale = _make_ap_bridge(
"old@m.example", accessed_days_ago=BRIDGE_STALE_DAYS + 1
)
Post.objects.create(
profile=stale.profile,
post_id="2025-05-01T10:00:00+00:00",
content="Bye",
created_at="2025-05-01T10:00:00+00:00",
)
active = _make_ap_bridge("new@m.example", accessed_days_ago=1)
# When: The cleanup task runs
deleted = _cleanup_stale_bridges_impl()
# Then: The stale bridge, its profile and posts are gone
self.assertEqual(deleted, 1)
self.assertFalse(BridgedActivityPubAccount.objects.filter(pk=stale.pk).exists())
self.assertFalse(
Profile.objects.filter(feed=activitypub_feed_url("old@m.example")).exists()
)
self.assertEqual(Post.objects.count(), 0)
# And: The active bridge survives
self.assertTrue(BridgedActivityPubAccount.objects.filter(pk=active.pk).exists())
def test_stale_rss_bridges_are_deleted_too(self):
# Given: A stale RSS bridge
_make_rss_bridge(
"https://blog.example/feed.xml",
accessed_days_ago=BRIDGE_STALE_DAYS + 1,
)
# When: The cleanup task runs
deleted = _cleanup_stale_bridges_impl()
# Then: It is deleted
self.assertEqual(deleted, 1)
self.assertEqual(BridgedRssFeed.objects.count(), 0)
def test_nothing_to_delete_returns_zero(self):
# Given: Only recently accessed bridges
_make_ap_bridge("alice@m.example", accessed_days_ago=1)
# When: The cleanup task runs
deleted = _cleanup_stale_bridges_impl()
# Then: Nothing is deleted
self.assertEqual(deleted, 0)
self.assertEqual(BridgedActivityPubAccount.objects.count(), 1)

383
app/bridge/test_views.py Normal file
View file

@ -0,0 +1,383 @@
from datetime import timedelta
from unittest.mock import patch
from django.test import TestCase
from django.utils import timezone
from app.bridge.fetching import BridgeError
from app.bridge.models import (
BridgedActivityPubAccount,
BridgedRssFeed,
activitypub_feed_url,
rss_feed_url,
)
from app.feeds.models import Profile
AP_FEED_PATH = "/bridge/activitypub/@alice@m.example/"
RSS_SOURCE_URL = "https://blog.example/feed.xml"
def _ap_data(posts=None):
return {
"actor_url": "https://m.example/users/alice",
"outbox_url": "https://m.example/users/alice/outbox",
"metadata": {
"title": "Alice",
"nick": "alice",
"description": "Bio",
"avatar": "https://m.example/avatar.png",
"link": "https://m.example/@alice",
},
"posts": posts
if posts is not None
else [
{
"id": "2025-05-01T10:00:00+00:00",
"content": "Hello from the fediverse",
"tags": "",
"language": "",
}
],
}
def _rss_data():
return {
"metadata": {
"title": "My Blog",
"nick": "My_Blog",
"description": "Notes",
"avatar": "",
"link": "https://blog.example",
"language": "en",
},
"posts": [
{
"id": "2025-05-01T10:00:00+00:00",
"content": "*** First article\n\nBody\n\n[[https://blog.example/first]]",
"tags": "",
"language": "",
}
],
}
class ActivityPubBridgeFeedViewTest(TestCase):
"""Test cases for the ActivityPub virtual feed endpoint."""
def test_first_get_registers_account_and_serves_org_file(self):
# Given: A reachable ActivityPub account not yet bridged
with patch(
"app.bridge.activitypub.fetch_account", return_value=_ap_data()
) as mock_fetch:
# When: The virtual feed is requested for the first time
response = self.client.get(AP_FEED_PATH)
# Then: The org file is served as plain text
self.assertEqual(response.status_code, 200)
self.assertTrue(response["Content-Type"].startswith("text/plain"))
content = response.content.decode()
self.assertIn("#+TITLE: Alice", content)
self.assertIn("#+NICK: alice", content)
self.assertIn("** 2025-05-01T10:00:00+00:00", content)
self.assertIn("Hello from the fediverse", content)
# And: The bridge and its profile are stored
mock_fetch.assert_called_once_with("alice", "m.example")
bridge = BridgedActivityPubAccount.objects.get(handle="alice@m.example")
self.assertEqual(bridge.profile.feed, activitypub_feed_url("alice@m.example"))
def test_second_get_is_served_from_database_without_network(self):
# Given: An already bridged account
with patch(
"app.bridge.activitypub.fetch_account", return_value=_ap_data()
) as mock_fetch:
self.client.get(AP_FEED_PATH)
# When: The feed is requested again shortly after
response = self.client.get(AP_FEED_PATH)
# Then: The response is correct and no second fetch happened
self.assertEqual(response.status_code, 200)
self.assertIn("Hello from the fediverse", response.content.decode())
self.assertEqual(mock_fetch.call_count, 1)
def test_non_canonical_handles_redirect_to_canonical_url(self):
# Given: Handle spellings with capitals or without the @ prefix
for path in (
"/bridge/activitypub/@Alice@M.Example/",
"/bridge/activitypub/alice@m.example/",
):
# When: The non-canonical URL is requested
response = self.client.get(path)
# Then: A permanent redirect points to the canonical URL
self.assertEqual(response.status_code, 301, path)
self.assertEqual(response["Location"], AP_FEED_PATH)
def test_invalid_handle_returns_400(self):
# Given: A handle without an instance part
# When: The feed is requested
response = self.client.get("/bridge/activitypub/@alice/")
# Then: The request is rejected
self.assertEqual(response.status_code, 400)
self.assertEqual(response.json()["type"], "Error")
def test_unreachable_account_returns_502(self):
# Given: An account whose instance cannot be reached
with patch(
"app.bridge.activitypub.fetch_account",
side_effect=BridgeError("connection refused"),
):
# When: The feed is requested
response = self.client.get(AP_FEED_PATH)
# Then: A bad gateway error with the envelope format is returned
self.assertEqual(response.status_code, 502)
body = response.json()
self.assertEqual(body["type"], "Error")
self.assertIsNone(body["data"])
# And: Nothing was stored
self.assertEqual(BridgedActivityPubAccount.objects.count(), 0)
def test_stale_bridge_is_refreshed_inline_on_access(self):
# Given: A bridged account whose data is older than a day
with patch("app.bridge.activitypub.fetch_account", return_value=_ap_data()):
self.client.get(AP_FEED_PATH)
bridge = BridgedActivityPubAccount.objects.get(handle="alice@m.example")
bridge.last_refreshed_at = timezone.now() - timedelta(days=2)
bridge.save()
new_posts = _ap_data(
posts=[
{
"id": "2025-05-05T10:00:00+00:00",
"content": "Fresh post",
"tags": "",
"language": "",
}
]
)
# When: The feed is requested again
with patch(
"app.bridge.activitypub.fetch_account", return_value=new_posts
) as mock_fetch:
response = self.client.get(AP_FEED_PATH)
# Then: The origin was fetched again and the new post is served
mock_fetch.assert_called_once()
self.assertIn("Fresh post", response.content.decode())
bridge.refresh_from_db()
self.assertGreater(
bridge.last_refreshed_at, timezone.now() - timedelta(minutes=1)
)
def test_failed_inline_refresh_serves_stored_copy(self):
# Given: A stale bridge whose origin is now unreachable
with patch("app.bridge.activitypub.fetch_account", return_value=_ap_data()):
self.client.get(AP_FEED_PATH)
bridge = BridgedActivityPubAccount.objects.get(handle="alice@m.example")
bridge.last_refreshed_at = timezone.now() - timedelta(days=2)
bridge.save()
# When: The feed is requested while the origin fails
with patch(
"app.bridge.activitypub.fetch_account",
side_effect=BridgeError("down"),
):
response = self.client.get(AP_FEED_PATH)
# Then: The stored copy is served anyway
self.assertEqual(response.status_code, 200)
self.assertIn("Hello from the fediverse", response.content.decode())
def test_bridged_profile_is_available_through_profile_endpoint(self):
# Given: A bridged account
with patch("app.bridge.activitypub.fetch_account", return_value=_ap_data()):
self.client.get(AP_FEED_PATH)
# When: The existing /profile/ endpoint is queried with the bridge URL
feed_url = activitypub_feed_url("alice@m.example")
response = self.client.get("/profile/", {"feed": feed_url})
# Then: The bridged profile is returned like any other profile
self.assertEqual(response.status_code, 200)
self.assertEqual(response.json()["type"], "Success")
class ActivityPubBridgeListViewTest(TestCase):
"""Test cases for the bridged accounts listing."""
def test_lists_bridged_accounts_with_feed_urls(self):
# Given: A bridged account
profile = Profile.objects.create(
feed=activitypub_feed_url("alice@m.example"),
title="Alice",
nick="alice",
version="v1",
)
BridgedActivityPubAccount.objects.create(
handle="alice@m.example",
actor_url="https://m.example/users/alice",
profile=profile,
last_accessed_at=timezone.now(),
)
# When: The list is requested
response = self.client.get("/bridge/activitypub/")
# Then: The account and its virtual feed URL are listed
self.assertEqual(response.status_code, 200)
body = response.json()
self.assertEqual(body["type"], "Success")
self.assertEqual(
body["data"],
[
{
"handle": "@alice@m.example",
"feed": activitypub_feed_url("alice@m.example"),
}
],
)
class RssBridgeViewTest(TestCase):
"""Test cases for the RSS bridge endpoint."""
def test_first_get_registers_feed_and_serves_org_file(self):
# Given: A reachable RSS feed not yet bridged
with patch(
"app.bridge.rss.fetch_rss_feed", return_value=_rss_data()
) as mock_fetch:
# When: The virtual feed is requested for the first time
response = self.client.get("/bridge/rss/", {"url": RSS_SOURCE_URL})
# Then: The org file is served as plain text
self.assertEqual(response.status_code, 200)
self.assertTrue(response["Content-Type"].startswith("text/plain"))
content = response.content.decode()
self.assertIn("#+TITLE: My Blog", content)
self.assertIn("#+NICK: My_Blog", content)
self.assertIn("*** First article", content)
# And: The bridge is stored
mock_fetch.assert_called_once_with(RSS_SOURCE_URL)
self.assertTrue(
BridgedRssFeed.objects.filter(source_url=RSS_SOURCE_URL).exists()
)
def test_second_get_is_served_from_database_without_network(self):
# Given: An already bridged RSS feed
with patch(
"app.bridge.rss.fetch_rss_feed", return_value=_rss_data()
) as mock_fetch:
self.client.get("/bridge/rss/", {"url": RSS_SOURCE_URL})
# When: The feed is requested again shortly after
response = self.client.get("/bridge/rss/", {"url": RSS_SOURCE_URL})
# Then: The response is correct and no second fetch happened
self.assertEqual(response.status_code, 200)
self.assertEqual(mock_fetch.call_count, 1)
def test_without_url_parameter_lists_bridged_feeds(self):
# Given: A bridged RSS feed
profile = Profile.objects.create(
feed=rss_feed_url(RSS_SOURCE_URL),
title="My Blog",
nick="My_Blog",
version="v1",
)
BridgedRssFeed.objects.create(
source_url=RSS_SOURCE_URL,
profile=profile,
last_accessed_at=timezone.now(),
)
# When: The endpoint is requested without url
response = self.client.get("/bridge/rss/")
# Then: The bridged feeds are listed
self.assertEqual(response.status_code, 200)
body = response.json()
self.assertEqual(
body["data"],
[{"url": RSS_SOURCE_URL, "feed": rss_feed_url(RSS_SOURCE_URL)}],
)
def test_invalid_url_parameter_returns_400(self):
# Given: Invalid url parameters
too_long = "https://example.com/" + "a" * 500
for bad_url in ("ftp://example.com/feed", "not-a-url", too_long):
# When: The endpoint is requested
response = self.client.get("/bridge/rss/", {"url": bad_url})
# Then: The request is rejected
self.assertEqual(response.status_code, 400, bad_url)
self.assertEqual(response.json()["type"], "Error")
def test_unreachable_feed_returns_502(self):
# Given: A feed URL that cannot be fetched
with patch(
"app.bridge.rss.fetch_rss_feed",
side_effect=BridgeError("timeout"),
):
# When: The endpoint is requested
response = self.client.get("/bridge/rss/", {"url": RSS_SOURCE_URL})
# Then: A bad gateway error is returned and nothing is stored
self.assertEqual(response.status_code, 502)
self.assertEqual(BridgedRssFeed.objects.count(), 0)
class BridgeIndexViewTest(TestCase):
"""Test cases for the bridge index endpoint."""
def test_index_exposes_bridge_links(self):
# When: The bridge index is requested
response = self.client.get("/bridge/")
# Then: Both bridge types are discoverable
self.assertEqual(response.status_code, 200)
links = response.json()["_links"]
self.assertIn("activitypub-feed", links)
self.assertIn("rss-feed", links)
def test_root_endpoint_links_to_bridges(self):
# When: The relay root is requested
response = self.client.get("/")
# Then: The bridge endpoints are listed
links = response.json()["_links"]
self.assertIn("bridge", links)
self.assertIn("bridge-activitypub", links)
self.assertIn("bridge-rss", links)
class LastAccessedTrackingTest(TestCase):
"""Test cases for access tracking used by refresh and cleanup."""
def test_old_last_accessed_is_updated_on_get(self):
# Given: A bridge last accessed two days ago
profile = Profile.objects.create(
feed=activitypub_feed_url("alice@m.example"),
title="Alice",
nick="alice",
version="v1",
)
old_access = timezone.now() - timedelta(days=2)
bridge = BridgedActivityPubAccount.objects.create(
handle="alice@m.example",
actor_url="https://m.example/users/alice",
profile=profile,
last_accessed_at=old_access,
last_refreshed_at=timezone.now(),
)
# When: The feed is requested
self.client.get(AP_FEED_PATH)
# Then: last_accessed_at moves forward
bridge.refresh_from_db()
self.assertGreater(bridge.last_accessed_at, old_access)

23
app/bridge/urls.py Normal file
View file

@ -0,0 +1,23 @@
from django.urls import path, re_path
from .views import (
ActivityPubBridgeFeedView,
ActivityPubBridgeListView,
BridgeIndexView,
RssBridgeView,
)
urlpatterns = [
path("", BridgeIndexView.as_view(), name="bridge-index"),
path(
"activitypub/",
ActivityPubBridgeListView.as_view(),
name="bridge-activitypub-list",
),
re_path(
r"^activitypub/(?P<handle>[^/]+)/$",
ActivityPubBridgeFeedView.as_view(),
name="bridge-activitypub-feed",
),
path("rss/", RssBridgeView.as_view(), name="bridge-rss"),
]

207
app/bridge/views.py Normal file
View file

@ -0,0 +1,207 @@
"""
Bridge endpoints: expose ActivityPub accounts and RSS/Atom feeds as
virtual social.org files that any Org Social client can #+FOLLOW:.
Registration is implicit: the first GET of an unknown account fetches
it from the origin and stores it. Later requests are served from the
database; the periodic refresh task keeps active bridges up to date,
and a request older than REFRESH_ON_ACCESS triggers an inline refresh.
"""
import logging
from datetime import timedelta
from django.http import HttpResponse, HttpResponsePermanentRedirect
from django.utils import timezone
from rest_framework import status
from rest_framework.response import Response
from rest_framework.views import APIView
from . import store
from .activitypub import normalize_handle
from .fetching import BridgeError
from .models import BridgedActivityPubAccount, BridgedRssFeed
from .org_renderer import render_profile_org
logger = logging.getLogger(__name__)
# A GET on data older than this triggers a synchronous refresh, which
# also reactivates bridges that fell out of the periodic refresh window
REFRESH_ON_ACCESS = timedelta(hours=24)
# last_accessed_at is written at most once per hour
ACCESS_UPDATE_THROTTLE = timedelta(hours=1)
MAX_SOURCE_URL_LENGTH = 500
def _error_response(errors, http_status):
return Response(
{"type": "Error", "errors": errors, "data": None}, status=http_status
)
def _serve_bridge(bridge, refresh_callable):
"""Common flow: refresh if stale, track access, render the org file."""
now = timezone.now()
if (
bridge.last_refreshed_at is None
or now - bridge.last_refreshed_at > REFRESH_ON_ACCESS
):
try:
refresh_callable(bridge)
except BridgeError as e:
# Serve the stored copy; the origin may recover later
logger.warning(f"Inline refresh failed for {bridge}: {e}")
if now - bridge.last_accessed_at > ACCESS_UPDATE_THROTTLE:
bridge.last_accessed_at = now
bridge.save(update_fields=["last_accessed_at"])
posts = bridge.profile.posts.order_by("created_at")
content = render_profile_org(bridge.profile, posts)
return HttpResponse(content, content_type="text/plain; charset=utf-8")
class BridgeIndexView(APIView):
"""Information about the available bridges."""
def get(self, request):
return Response(
{
"type": "Success",
"errors": [],
"data": {
"description": (
"Bridges expose external accounts as virtual "
"social.org feeds that can be followed with #+FOLLOW:"
),
},
"_links": {
"self": {"href": "/bridge/", "method": "GET"},
"activitypub-feed": {
"href": "/bridge/activitypub/@{user}@{instance}/",
"method": "GET",
"templated": True,
},
"activitypub-list": {
"href": "/bridge/activitypub/",
"method": "GET",
},
"rss-feed": {
"href": "/bridge/rss/?url={feed_url}",
"method": "GET",
"templated": True,
},
"rss-list": {"href": "/bridge/rss/", "method": "GET"},
},
}
)
class ActivityPubBridgeListView(APIView):
"""List bridged ActivityPub accounts."""
def get(self, request):
data = [
{"handle": f"@{bridge.handle}", "feed": bridge.feed_url}
for bridge in BridgedActivityPubAccount.objects.all()
]
return Response(
{
"type": "Success",
"errors": [],
"data": data,
"_links": {
"self": {"href": "/bridge/activitypub/", "method": "GET"},
},
}
)
class ActivityPubBridgeFeedView(APIView):
"""Serve an ActivityPub account as a virtual social.org file."""
def get(self, request, handle):
normalized = normalize_handle(handle)
if normalized is None:
return _error_response(
["Invalid handle. Expected format: @user@instance"],
status.HTTP_400_BAD_REQUEST,
)
# Redirect non-canonical spellings so each account has one URL
if handle != f"@{normalized}":
return HttpResponsePermanentRedirect(f"/bridge/activitypub/@{normalized}/")
bridge = (
BridgedActivityPubAccount.objects.select_related("profile")
.filter(handle=normalized)
.first()
)
if bridge is None:
try:
bridge = store.create_activitypub_bridge(normalized)
except BridgeError as e:
logger.warning(f"Could not bridge @{normalized}: {e}")
return _error_response(
[f"Could not fetch ActivityPub account: {e}"],
status.HTTP_502_BAD_GATEWAY,
)
return _serve_bridge(bridge, store.refresh_activitypub_bridge)
class RssBridgeView(APIView):
"""
With ?url= serves an RSS/Atom feed as a virtual social.org file;
without it, lists the bridged RSS feeds.
"""
def get(self, request):
source_url = request.query_params.get("url")
if source_url is None:
data = [
{"url": bridge.source_url, "feed": bridge.feed_url}
for bridge in BridgedRssFeed.objects.all()
]
return Response(
{
"type": "Success",
"errors": [],
"data": data,
"_links": {
"self": {"href": "/bridge/rss/", "method": "GET"},
},
}
)
source_url = source_url.strip()
if (
not source_url.startswith(("http://", "https://"))
or len(source_url) > MAX_SOURCE_URL_LENGTH
):
return _error_response(
["Invalid url parameter. Expected an http(s) feed URL"],
status.HTTP_400_BAD_REQUEST,
)
bridge = (
BridgedRssFeed.objects.select_related("profile")
.filter(source_url=source_url)
.first()
)
if bridge is None:
try:
bridge = store.create_rss_bridge(source_url)
except BridgeError as e:
logger.warning(f"Could not bridge RSS feed {source_url}: {e}")
return _error_response(
[f"Could not fetch RSS feed: {e}"],
status.HTTP_502_BAD_GATEWAY,
)
return _serve_bridge(bridge, store.refresh_rss_bridge)

0
app/feedcontent/__init__.py Normal file
View file

6
app/feedcontent/apps.py Normal file
View file

@ -0,0 +1,6 @@
from django.apps import AppConfig
class FeedcontentConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "app.feedcontent"

204
app/feedcontent/tests.py Normal file
View file

@ -0,0 +1,204 @@
from django.test import TestCase
from rest_framework.test import APIClient
from unittest.mock import patch, Mock
from app.feeds.models import Profile
import requests
class FeedContentViewTests(TestCase):
def setUp(self):
"""Set up test fixtures."""
self.client = APIClient()
self.profile = Profile.objects.create(
feed="https://example.com/social.org",
title="Test Profile",
nick="testuser",
)
def test_get_feed_content_success(self):
"""Test successfully fetching feed content"""
# Given: A registered feed whose server returns valid Org Social content
mock_content = """#+TITLE: My Social Feed
#+AUTHOR: John Doe
* 2025-02-05T10:00:00+0100
:PROPERTIES:
:ID: 2025-02-05T10:00:00+0100
:END:
Hello, world! This is my first post.
"""
with patch("app.feedcontent.views.requests.get") as mock_get:
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = mock_content.encode("utf-8")
mock_response.raise_for_status = Mock()
mock_get.return_value = mock_response
# When: The feed content endpoint is requested
response = self.client.get(
"/feed-content/", {"feed": "https://example.com/social.org"}
)
# Then: The content is returned verbatim with HATEOAS links
self.assertEqual(response.status_code, 200)
self.assertEqual(response.data["type"], "Success")
self.assertEqual(response.data["data"]["content"], mock_content)
self.assertIn("_links", response.data)
self.assertIn("self", response.data["_links"])
def test_get_feed_content_missing_parameter(self):
"""Test error when feed parameter is missing"""
# Given: No feed parameter
# When: The feed content endpoint is requested
response = self.client.get("/feed-content/")
# Then: A 400 error explains the missing parameter
self.assertEqual(response.status_code, 400)
self.assertEqual(response.data["type"], "Error")
self.assertIn("Feed URL parameter is required", response.data["errors"])
def test_get_feed_content_empty_parameter(self):
"""Test error when feed parameter is empty"""
# Given: A blank feed parameter
# When: The feed content endpoint is requested
response = self.client.get("/feed-content/", {"feed": " "})
# Then: A 400 error explains the missing parameter
self.assertEqual(response.status_code, 400)
self.assertEqual(response.data["type"], "Error")
self.assertIn("Feed URL parameter is required", response.data["errors"])
def test_get_feed_content_feed_not_found(self):
"""Test error when feed is not registered in relay"""
# Given: A feed URL that is not registered in the relay
# When: The feed content endpoint is requested
response = self.client.get(
"/feed-content/", {"feed": "https://unknown.com/social.org"}
)
# Then: A 404 error explains the feed is unknown
self.assertEqual(response.status_code, 404)
self.assertEqual(response.data["type"], "Error")
self.assertIn("Feed not found in relay", response.data["errors"])
def test_get_feed_content_timeout(self):
"""Test error when feed server times out"""
# Given: A registered feed whose server times out
with patch("app.feedcontent.views.requests.get") as mock_get:
mock_get.side_effect = requests.exceptions.Timeout()
# When: The feed content endpoint is requested
response = self.client.get(
"/feed-content/", {"feed": "https://example.com/social.org"}
)
# Then: A 502 error reports the timeout
self.assertEqual(response.status_code, 502)
self.assertEqual(response.data["type"], "Error")
self.assertIn("Request timeout", response.data["errors"][0])
def test_get_feed_content_connection_error(self):
"""Test error when connection to feed server fails"""
# Given: A registered feed whose server refuses the connection
with patch("app.feedcontent.views.requests.get") as mock_get:
mock_get.side_effect = requests.exceptions.ConnectionError()
# When: The feed content endpoint is requested
response = self.client.get(
"/feed-content/", {"feed": "https://example.com/social.org"}
)
# Then: A 502 error reports the connection failure
self.assertEqual(response.status_code, 502)
self.assertEqual(response.data["type"], "Error")
self.assertIn("Connection error", response.data["errors"][0])
def test_get_feed_content_http_error(self):
"""Test error when feed server returns HTTP error"""
# Given: A registered feed whose server answers with an HTTP error
with patch("app.feedcontent.views.requests.get") as mock_get:
mock_response = Mock()
mock_response.raise_for_status.side_effect = requests.exceptions.HTTPError(
"404 Not Found"
)
mock_get.return_value = mock_response
# When: The feed content endpoint is requested
response = self.client.get(
"/feed-content/", {"feed": "https://example.com/social.org"}
)
# Then: A 502 error reports the upstream HTTP error
self.assertEqual(response.status_code, 502)
self.assertEqual(response.data["type"], "Error")
self.assertIn("HTTP error", response.data["errors"][0])
def test_get_feed_content_unicode(self):
"""Test fetching feed content with unicode characters"""
# Given: A registered feed whose content includes unicode and emojis
mock_content = """#+TITLE: Mi Feed Social
#+AUTHOR: José García
* 2025-02-05T10:00:00+0100
:PROPERTIES:
:ID: 2025-02-05T10:00:00+0100
:END:
¡Hola mundo! Este es mi primer post con émojis 🚀 ❤️
"""
with patch("app.feedcontent.views.requests.get") as mock_get:
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = mock_content.encode("utf-8")
mock_response.raise_for_status = Mock()
mock_get.return_value = mock_response
# When: The feed content endpoint is requested
response = self.client.get(
"/feed-content/", {"feed": "https://example.com/social.org"}
)
# Then: The unicode content survives the round trip intact
self.assertEqual(response.status_code, 200)
self.assertEqual(response.data["type"], "Success")
self.assertEqual(response.data["data"]["content"], mock_content)
self.assertIn("🚀", response.data["data"]["content"])
self.assertIn("❤️", response.data["data"]["content"])
def test_get_feed_content_preserves_formatting(self):
"""Test that feed content preserves whitespace and formatting"""
# Given: A registered feed whose content mixes blank lines, spaces and tabs
mock_content = """#+TITLE: Test Feed
* 2025-02-05T10:00:00+0100
:PROPERTIES:
:ID: 2025-02-05T10:00:00+0100
:END:
Line 1
Line 2 with multiple spaces
Line 3 with tab
"""
with patch("app.feedcontent.views.requests.get") as mock_get:
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = mock_content.encode("utf-8")
mock_response.raise_for_status = Mock()
mock_get.return_value = mock_response
# When: The feed content endpoint is requested
response = self.client.get(
"/feed-content/", {"feed": "https://example.com/social.org"}
)
# Then: The content is exactly as provided, preserving all whitespace
self.assertEqual(response.status_code, 200)
self.assertEqual(response.data["data"]["content"], mock_content)

6
app/feedcontent/urls.py Normal file
View file

@ -0,0 +1,6 @@
from django.urls import path
from .views import FeedContentView
urlpatterns = [
path("", FeedContentView.as_view(), name="feed-content"),
]

106
app/feedcontent/views.py Normal file
View file

@ -0,0 +1,106 @@
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework import status
import logging
import requests
from app.feeds.models import Profile
logger = logging.getLogger(__name__)
class FeedContentView(APIView):
"""Get raw content of an Org Social feed file"""
def get(self, request):
feed_url = request.query_params.get("feed")
if not feed_url or not feed_url.strip():
return Response(
{
"type": "Error",
"errors": ["Feed URL parameter is required"],
"data": None,
},
status=status.HTTP_400_BAD_REQUEST,
)
feed_url = feed_url.strip()
# Check if the feed is registered in the relay
try:
Profile.objects.get(feed=feed_url)
except Profile.DoesNotExist:
return Response(
{
"type": "Error",
"errors": ["Feed not found in relay"],
"data": None,
},
status=status.HTTP_404_NOT_FOUND,
)
# Fetch the feed content from the source URL
try:
response = requests.get(feed_url, timeout=10)
response.raise_for_status()
# Decode content as UTF-8
content = response.content.decode("utf-8")
# URL encode the feed_url for the self link
from urllib.parse import quote
encoded_feed_url = quote(feed_url, safe="")
response_data = {
"type": "Success",
"errors": [],
"data": {"content": content},
"_links": {
"self": {
"href": f"/feed-content/?feed={encoded_feed_url}",
"method": "GET",
}
},
}
return Response(response_data, status=status.HTTP_200_OK)
except requests.exceptions.Timeout:
return Response(
{
"type": "Error",
"errors": ["Request timeout: Feed server did not respond in time"],
"data": None,
},
status=status.HTTP_502_BAD_GATEWAY,
)
except requests.exceptions.ConnectionError:
return Response(
{
"type": "Error",
"errors": ["Connection error: Could not reach feed server"],
"data": None,
},
status=status.HTTP_502_BAD_GATEWAY,
)
except requests.exceptions.HTTPError as e:
return Response(
{
"type": "Error",
"errors": [f"HTTP error: {e}"],
"data": None,
},
status=status.HTTP_502_BAD_GATEWAY,
)
except Exception as e:
logger.error(f"Failed to fetch feed content from {feed_url}: {e}")
return Response(
{
"type": "Error",
"errors": ["Failed to fetch feed content"],
"data": None,
},
status=status.HTTP_502_BAD_GATEWAY,
)

49
app/feeds/cors_middleware.py Normal file
View file

@ -0,0 +1,49 @@
"""
Middleware for adding CORS headers to all responses.
"""
import logging
logger = logging.getLogger(__name__)
class CORSMiddleware:
"""
Middleware that adds CORS headers to all responses.
This middleware ensures that:
- All API responses are accessible from any origin (Access-Control-Allow-Origin: *)
- Preflight OPTIONS requests are handled correctly
- Common HTTP methods and headers are allowed
"""
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
# Handle preflight OPTIONS requests
if request.method == "OPTIONS":
response = self._create_options_response()
else:
# Get the response from the view
response = self.get_response(request)
# Add CORS headers to all responses
response["Access-Control-Allow-Origin"] = "*"
response["Access-Control-Allow-Methods"] = (
"GET, POST, PUT, PATCH, DELETE, OPTIONS"
)
response["Access-Control-Allow-Headers"] = (
"Content-Type, Authorization, X-Requested-With"
)
response["Access-Control-Max-Age"] = "86400" # 24 hours
return response
def _create_options_response(self):
"""Create a response for OPTIONS preflight requests"""
from django.http import HttpResponse
response = HttpResponse()
response.status_code = 200
return response

58
app/feeds/middleware.py Normal file
View file

@ -0,0 +1,58 @@
"""
Middleware for adding global HTTP caching headers to all responses.
"""
import logging
from django.core.cache import cache
from app.feeds.models import RelayMetadata
logger = logging.getLogger(__name__)
class RelayMetadataMiddleware:
"""
Middleware that adds global ETag and Last-Modified headers to all responses.
This middleware ensures that:
- All API responses get consistent caching headers
- Headers are added even when serving cached responses
- The ETag and Last-Modified come from RelayMetadata (updated by scan_feeds)
- All clients see the same cache version across all endpoints
- Metadata is cached to avoid DB queries on every request
"""
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
# Get the response from the view
response = self.get_response(request)
# Add global relay headers to successful responses
# Only add to 200-299 status codes (successful responses)
if 200 <= response.status_code < 300:
try:
# Try to get metadata from cache first
cache_key = "relay_metadata_headers"
cached_data = cache.get(cache_key)
if cached_data is None:
# If not in cache, query database
etag, last_modified = RelayMetadata.get_global_metadata()
cached_data = (etag, last_modified)
# Cache permanently until scan_feeds invalidates it
cache.set(cache_key, cached_data, timeout=None)
logger.debug("Cached relay metadata from database")
else:
logger.debug("Using cached relay metadata")
etag, last_modified = cached_data
response["ETag"] = f'"{etag}"'
response["Last-Modified"] = last_modified.strftime(
"%a, %d %b %Y %H:%M:%S GMT"
)
except Exception as e:
# Log error but don't fail the request
logger.warning(f"Failed to add relay metadata headers: {e}")
return response

40
app/feeds/migrations/0007_relaymetadata.py Normal file
View file

@ -0,0 +1,40 @@
# Generated by Django 5.2.7 on 2025-11-05 13:49
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("feeds", "0006_feed_last_successful_fetch"),
]
operations = [
migrations.CreateModel(
name="RelayMetadata",
fields=[
(
"key",
models.CharField(
max_length=50, primary_key=True, serialize=False, unique=True
),
),
(
"etag",
models.CharField(
help_text="Global ETag for HTTP caching", max_length=16
),
),
(
"last_modified",
models.DateTimeField(
help_text="Global Last-Modified timestamp for HTTP caching"
),
),
("updated_at", models.DateTimeField(auto_now=True)),
],
options={
"verbose_name": "Relay Metadata",
"verbose_name_plural": "Relay Metadata",
},
),
]

21
app/feeds/migrations/0008_post_include.py Normal file
View file

@ -0,0 +1,21 @@
# Generated by Django 5.2.8 on 2025-11-16 17:34
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("feeds", "0007_relaymetadata"),
]
operations = [
migrations.AddField(
model_name="post",
name="include",
field=models.CharField(
blank=True,
help_text="Post being boosted/shared (URL#ID format)",
max_length=300,
),
),
]

19
app/feeds/migrations/0009_alter_post_created_at.py Normal file
View file

@ -0,0 +1,19 @@
# Generated by Django 5.2.6 on 2025-12-25 09:01
import django.utils.timezone
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('feeds', '0008_post_include'),
]
operations = [
migrations.AlterField(
model_name='post',
name='created_at',
field=models.DateTimeField(default=django.utils.timezone.now, help_text='Creation timestamp, parsed from post_id when available'),
),
]

33
app/feeds/migrations/0010_profile_birthday_profile_language_profile_location_and_more.py Normal file
View file

@ -0,0 +1,33 @@
# Generated by Django 5.2.6 on 2026-01-05 12:49
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('feeds', '0009_alter_post_created_at'),
]
operations = [
migrations.AddField(
model_name='profile',
name='birthday',
field=models.DateField(blank=True, help_text='User birthday in YYYY-MM-DD format', null=True),
),
migrations.AddField(
model_name='profile',
name='language',
field=models.CharField(blank=True, help_text='Space-separated language codes (ISO 639-1)', max_length=100),
),
migrations.AddField(
model_name='profile',
name='location',
field=models.CharField(blank=True, help_text='User location (city, country, etc.)', max_length=200),
),
migrations.AddField(
model_name='profile',
name='pinned',
field=models.CharField(blank=True, help_text='Pinned post ID (timestamp)', max_length=50),
),
]

31
app/feeds/migrations/0011_outgoingwebmention.py Normal file
View file

@ -0,0 +1,31 @@
# Generated by Django 6.0.7 on 2026-07-16 08:13
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('feeds', '0010_profile_birthday_profile_language_profile_location_and_more'),
]
operations = [
migrations.CreateModel(
name='OutgoingWebmention',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('source', models.URLField(help_text='Post URL (feed#post_id) containing the link', max_length=500)),
('target', models.URLField(help_text='External URL the post links to', max_length=500)),
('endpoint', models.URLField(blank=True, help_text='Discovered Webmention endpoint', max_length=500)),
('status', models.CharField(choices=[('pending', 'Pending'), ('sent', 'Sent'), ('failed', 'Failed'), ('no_endpoint', 'No endpoint')], default='pending', max_length=20)),
('attempts', models.PositiveSmallIntegerField(default=0)),
('response_code', models.IntegerField(blank=True, help_text='HTTP status of the last delivery attempt', null=True)),
('last_attempt_at', models.DateTimeField(blank=True, null=True)),
('created_at', models.DateTimeField(auto_now_add=True)),
],
options={
'ordering': ['created_at'],
'unique_together': {('source', 'target')},
},
),
]

24
app/feeds/migrations/0012_delete_bridge_feeds.py Normal file
View file

@ -0,0 +1,24 @@
from django.db import migrations
from django.db.models import Q
def delete_bridge_feeds(apps, schema_editor):
"""
Bridge virtual feeds are a connection helper, not real accounts.
Remove any that were registered before they were excluded from
registration and discovery.
"""
Feed = apps.get_model("feeds", "Feed")
Feed.objects.filter(
Q(url__contains="/bridge/activitypub/") | Q(url__contains="/bridge/rss/")
).delete()
class Migration(migrations.Migration):
dependencies = [
("feeds", "0011_outgoingwebmention"),
]
operations = [
migrations.RunPython(delete_bridge_feeds, migrations.RunPython.noop),
]

134
app/feeds/models.py
View file

@ -1,4 +1,8 @@
from django.db import models
from django.utils import timezone
import logging
logger = logging.getLogger(__name__)
class Profile(models.Model):
@ -15,6 +19,20 @@ class Profile(models.Model):
avatar = models.URLField(
blank=True, help_text="URL to avatar image (128x128px JPG/PNG)"
)
location = models.CharField(
max_length=200, blank=True, help_text="User location (city, country, etc.)"
)
birthday = models.DateField(
blank=True, null=True, help_text="User birthday in YYYY-MM-DD format"
)
language = models.CharField(
max_length=100,
blank=True,
help_text="Space-separated language codes (ISO 639-1)",
)
pinned = models.CharField(
max_length=50, blank=True, help_text="Pinned post ID (timestamp)"
)
version = models.CharField(
max_length=50,
blank=True,
@ -118,6 +136,11 @@ class Post(models.Model):
group = models.CharField(
max_length=100, blank=True, help_text="Group name (GROUP property)"
)
include = models.CharField(
max_length=300,
blank=True,
help_text="Post being boosted/shared (URL#ID format)",
)
# Poll related fields
poll_end = models.DateTimeField(
@ -125,7 +148,10 @@ class Post(models.Model):
)
# Metadata
created_at = models.DateTimeField(auto_now_add=True)
created_at = models.DateTimeField(
default=timezone.now,
help_text="Creation timestamp, parsed from post_id when available",
)
updated_at = models.DateTimeField(auto_now=True)
class Meta:
@ -143,6 +169,10 @@ class Post(models.Model):
def is_reply(self):
return bool(self.reply_to)
@property
def is_boost(self):
return bool(self.include)
class PollOption(models.Model):
"""
@ -225,3 +255,105 @@ class Feed(models.Model):
def __str__(self):
return self.url
class OutgoingWebmention(models.Model):
"""
Outbox of Webmentions sent by the relay on behalf of feed authors.
The (source, target) pair is unique, which guarantees each link found
in a post is notified at most once no matter how many times the feed
is rescanned.
"""
STATUS_PENDING = "pending"
STATUS_SENT = "sent"
STATUS_FAILED = "failed"
STATUS_NO_ENDPOINT = "no_endpoint"
STATUS_CHOICES = [
(STATUS_PENDING, "Pending"),
(STATUS_SENT, "Sent"),
(STATUS_FAILED, "Failed"),
(STATUS_NO_ENDPOINT, "No endpoint"),
]
source = models.URLField(
max_length=500, help_text="Post URL (feed#post_id) containing the link"
)
target = models.URLField(max_length=500, help_text="External URL the post links to")
endpoint = models.URLField(
max_length=500, blank=True, help_text="Discovered Webmention endpoint"
)
status = models.CharField(
max_length=20, choices=STATUS_CHOICES, default=STATUS_PENDING
)
attempts = models.PositiveSmallIntegerField(default=0)
response_code = models.IntegerField(
null=True, blank=True, help_text="HTTP status of the last delivery attempt"
)
last_attempt_at = models.DateTimeField(null=True, blank=True)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
unique_together = ["source", "target"]
ordering = ["created_at"]
def __str__(self):
return f"{self.source} -> {self.target} ({self.status})"
class RelayMetadata(models.Model):
"""
Global metadata for the relay - used for HTTP caching headers.
This model ensures all endpoints return consistent ETag and Last-Modified headers.
Updated by the scan_feeds task after each scan.
"""
key = models.CharField(max_length=50, unique=True, primary_key=True)
etag = models.CharField(max_length=16, help_text="Global ETag for HTTP caching")
last_modified = models.DateTimeField(
help_text="Global Last-Modified timestamp for HTTP caching"
)
updated_at = models.DateTimeField(auto_now=True)
class Meta:
verbose_name = "Relay Metadata"
verbose_name_plural = "Relay Metadata"
def __str__(self):
return f"RelayMetadata({self.key}): ETag={self.etag}"
@classmethod
def get_global_metadata(cls):
"""
Get or create the global relay metadata.
Returns a tuple of (etag, last_modified).
"""
import hashlib
from django.utils import timezone
metadata, created = cls.objects.get_or_create(
key="global",
defaults={
"etag": hashlib.md5(str(timezone.now()).encode()).hexdigest()[:8],
"last_modified": timezone.now(),
},
)
return metadata.etag, metadata.last_modified
@classmethod
def update_global_metadata(cls):
"""
Update the global ETag and Last-Modified.
Called by scan_feeds task after scanning all feeds.
"""
import hashlib
from django.utils import timezone
now = timezone.now()
etag = hashlib.md5(str(now.timestamp()).encode()).hexdigest()[:8]
cls.objects.update_or_create(
key="global", defaults={"etag": etag, "last_modified": now}
)
logger.info(f"Updated global relay metadata: ETag={etag}, Last-Modified={now}")

61
app/feeds/notification_publisher.py Normal file
View file

@ -0,0 +1,61 @@
"""
Notification publisher for real-time SSE notifications.
This module handles publishing notifications to Redis Pub/Sub channels
so that SSE clients can receive real-time updates.
"""
import json
import logging
import redis
from django.conf import settings
logger = logging.getLogger(__name__)
def get_redis_connection():
"""Get a Redis connection using Huey settings."""
try:
redis_host = settings.HUEY["connection"]["host"]
redis_port = settings.HUEY["connection"]["port"]
redis_db = settings.HUEY["connection"]["db"]
return redis.Redis(
host=redis_host, port=redis_port, db=redis_db, decode_responses=True
)
except Exception as e:
logger.error(f"Failed to create Redis connection: {e}")
return None
def publish_notification(target_feed_url, notification_type, post_url, **extra_data):
"""
Publish a notification to a feed's Redis Pub/Sub channel.
Args:
target_feed_url: URL of the feed that should receive the notification
notification_type: Type of notification ("mention", "reply", "reaction", "boost")
post_url: Full post URL (feed_url#post_id) that triggered the notification
**extra_data: Additional data to include in the notification (e.g., emoji, parent)
Returns:
bool: True if published successfully, False otherwise
"""
try:
r = get_redis_connection()
if not r:
return False
# Build notification payload
notification = {"type": notification_type, "post": post_url, **extra_data}
# Publish to the target feed's channel
channel = f"notifications:{target_feed_url}"
r.publish(channel, json.dumps(notification))
logger.debug(f"Published {notification_type} notification to {target_feed_url}")
return True
except Exception as e:
logger.error(f"Failed to publish notification: {e}")
return False

359
app/feeds/parser.py
View file

@ -1,7 +1,37 @@
import re
import requests
from datetime import datetime
from typing import Dict, Any, Tuple
from django.utils import timezone
import logging
logger = logging.getLogger(__name__)
# (connect, read) timeouts in seconds for fetching remote feeds.
# A short connect timeout drops unreachable hosts quickly (e.g. dead servers),
# while the read timeout stays generous enough for slow-but-alive servers.
FEED_FETCH_TIMEOUT = (3.05, 5)
def _sanitize_birthday(value: str) -> str:
"""Return the birthday only if it is a valid YYYY-MM-DD date, else "".
Some feeds use other date formats (e.g. "2003/06/17"). The Profile.birthday
DateField only accepts YYYY-MM-DD, so an invalid value would raise a
ValidationError and abort the whole feed scan. We drop the malformed value
instead of discarding an otherwise valid feed.
"""
value = (value or "").strip()
if not value:
return ""
try:
datetime.strptime(value, "%Y-%m-%d")
except ValueError:
logger.warning(
f"Ignoring invalid birthday value (expected YYYY-MM-DD): {value!r}"
)
return ""
return value
def _update_feed_last_successful_fetch(url: str):
@ -19,6 +49,211 @@ def _update_feed_last_successful_fetch(url: str):
pass
def _handle_feed_redirect(old_url: str, new_url: str):
"""
Handle feed URL redirect by updating or merging feeds.
If a feed redirects to a new URL:
1. Check if new URL already exists as a feed
2. If yes: merge data and delete old URL
3. If no: update old URL to new URL
Args:
old_url: Original URL that redirected
new_url: Final URL after redirect
"""
try:
from .models import Feed, Profile, Post, Follow, Mention, PollVote
from django.db import transaction
old_feed = Feed.objects.filter(url=old_url).first()
new_feed = Feed.objects.filter(url=new_url).first()
if old_feed and new_feed:
# Both URLs exist - merge them
logger.info(f"Feed redirect detected: {old_url} -> {new_url}")
logger.info("Both feeds exist. Merging old feed into new feed.")
with transaction.atomic():
# Get profiles for both feeds
old_profile = Profile.objects.filter(feed=old_url).first()
new_profile = Profile.objects.filter(feed=new_url).first()
if old_profile and new_profile:
# Merge profiles - keep the new one, migrate relationships
logger.info(
f"Merging profile data: {old_profile.nick} -> {new_profile.nick}"
)
# Migrate Follow relationships where old_profile is followed
follows_as_followed = Follow.objects.filter(followed=old_profile)
for follow in follows_as_followed:
# Check if this relationship already exists with new_profile
existing = Follow.objects.filter(
follower=follow.follower, followed=new_profile
).first()
if not existing:
# Update to point to new_profile
follow.followed = new_profile
try:
follow.save()
logger.debug(
f"Migrated follow relationship: {follow.follower.nick} -> {new_profile.nick}"
)
except Exception as e:
logger.warning(
f"Could not migrate follow relationship, deleting: {e}"
)
follow.delete()
else:
# Relationship already exists, delete duplicate
follow.delete()
logger.debug("Deleted duplicate follow relationship")
# Migrate Follow relationships where old_profile is follower
follows_as_follower = Follow.objects.filter(follower=old_profile)
for follow in follows_as_follower:
# Check if this relationship already exists with new_profile
existing = Follow.objects.filter(
follower=new_profile, followed=follow.followed
).first()
if not existing:
# Update to point to new_profile
follow.follower = new_profile
try:
follow.save()
logger.debug(
f"Migrated follow relationship: {new_profile.nick} -> {follow.followed.nick}"
)
except Exception as e:
logger.warning(
f"Could not migrate follow relationship, deleting: {e}"
)
follow.delete()
else:
# Relationship already exists, delete duplicate
follow.delete()
logger.debug("Deleted duplicate follow relationship")
# Migrate Mention relationships pointing to old_profile
# Mentions have unique constraint (post, mentioned_profile)
mentions_to_migrate = Mention.objects.filter(
mentioned_profile=old_profile
)
for mention in mentions_to_migrate:
# Check if this mention already exists with new_profile
existing = Mention.objects.filter(
post=mention.post, mentioned_profile=new_profile
).first()
if not existing:
# Update to point to new_profile
mention.mentioned_profile = new_profile
try:
mention.save()
logger.debug(
f"Migrated mention in post {mention.post.post_id}"
)
except Exception as e:
logger.warning(
f"Could not migrate mention, deleting: {e}"
)
mention.delete()
else:
# Mention already exists, delete duplicate
mention.delete()
logger.debug("Deleted duplicate mention")
# Migrate posts from old_profile to new_profile (avoid duplicates)
old_posts = Post.objects.filter(profile=old_profile)
for old_post in old_posts:
# Check if post already exists in new profile
existing_post = Post.objects.filter(
profile=new_profile, post_id=old_post.post_id
).first()
if not existing_post:
# Migrate post to new profile
old_post.profile = new_profile
old_post.save()
logger.debug(
f"Migrated post {old_post.post_id} to new profile"
)
else:
# Post already exists, handle poll votes carefully
# Get all poll votes pointing to old_post
poll_votes = PollVote.objects.filter(poll_post=old_post)
for poll_vote in poll_votes:
try:
# Check if this vote already exists for the existing_post
existing_vote = PollVote.objects.filter(
post=poll_vote.post, poll_post=existing_post
).first()
if not existing_vote:
# Update to point to existing_post
poll_vote.poll_post = existing_post
poll_vote.save()
logger.debug(
"Migrated poll vote to existing post"
)
else:
# Vote already exists, delete duplicate
poll_vote.delete()
logger.debug("Deleted duplicate poll vote")
except Exception as e:
# If there's any constraint error, just delete the vote
logger.warning(
f"Error migrating poll vote, deleting: {e}"
)
poll_vote.delete()
# Delete duplicate post
old_post.delete()
logger.debug(f"Removed duplicate post {old_post.post_id}")
# Delete old profile
old_profile.delete()
logger.info(f"Deleted old profile: {old_url}")
elif old_profile and not new_profile:
# Only old profile exists - update its feed URL
logger.info(f"Updating profile feed URL: {old_url} -> {new_url}")
old_profile.feed = new_url
old_profile.save()
# Delete the old feed
old_feed.delete()
logger.info(f"Deleted old feed: {old_url}")
elif old_feed and not new_feed:
# Only old URL exists - update it to new URL
logger.info(f"Feed redirect detected: {old_url} -> {new_url}")
logger.info(f"Updating feed URL to: {new_url}")
with transaction.atomic():
# Update the feed URL
old_feed.url = new_url
old_feed.save()
# Update all profiles pointing to old URL
profiles_updated = Profile.objects.filter(feed=old_url).update(
feed=new_url
)
logger.info(f"Updated {profiles_updated} profile(s) to new URL")
# If new_feed exists but not old_feed, nothing to do
# This can happen if the redirect was already processed
except Exception as e:
# Don't break parsing if redirect handling fails
logger.error(
f"Failed to handle redirect {old_url} -> {new_url}: {e}", exc_info=True
)
def parse_org_social(url: str) -> Dict[str, Any]:
"""
Parse an Org Social file from a URL and return structured data.
@ -30,9 +265,22 @@ def parse_org_social(url: str) -> Dict[str, Any]:
Dictionary containing parsed metadata and posts
"""
try:
response = requests.get(url, timeout=5)
response = requests.get(url, timeout=FEED_FETCH_TIMEOUT)
response.raise_for_status()
content = response.text
# Decode content as UTF-8 explicitly to avoid encoding issues
# when the server doesn't specify charset in Content-Type header
content = response.content.decode("utf-8")
# Check if URL was redirected
final_url = response.url
if final_url != url and response.history:
# URL was redirected - handle the redirect
logger.info(
f"Redirect detected: {url} -> {final_url} (status: {response.history[0].status_code})"
)
_handle_feed_redirect(url, final_url)
# Use final URL for further operations
url = final_url
# Update last_successful_fetch if we got a 200 response
if response.status_code == 200:
@ -48,6 +296,10 @@ def parse_org_social(url: str) -> Dict[str, Any]:
"nick": "",
"description": "",
"avatar": "",
"location": "",
"birthday": "",
"language": "",
"pinned": "",
"links": [],
"follows": [],
"contacts": [],
@ -76,6 +328,25 @@ def parse_org_social(url: str) -> Dict[str, Any]:
avatar_match = re.search(r"^\s*\#\+AVATAR:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["avatar"] = avatar_match.group(1).strip() if avatar_match else ""
# Parse new v1.6 fields
location_match = re.search(r"^\s*\#\+LOCATION:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["location"] = (
location_match.group(1).strip() if location_match else ""
)
birthday_match = re.search(r"^\s*\#\+BIRTHDAY:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["birthday"] = (
_sanitize_birthday(birthday_match.group(1)) if birthday_match else ""
)
language_match = re.search(r"^\s*\#\+LANGUAGE:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["language"] = (
language_match.group(1).strip() if language_match else ""
)
pinned_match = re.search(r"^\s*\#\+PINNED:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["pinned"] = pinned_match.group(1).strip() if pinned_match else ""
# Parse multiple values
result["metadata"]["links"] = [
match.group(1).strip()
@ -107,14 +378,18 @@ def parse_org_social(url: str) -> Dict[str, Any]:
# Split posts by ** headers (exactly 2 asterisks, not 3+)
# Use negative lookahead (?!\*) to ensure we don't match *** or ****
# Use ^ anchor to match ** only at start of line
post_pattern = r"\*\*(?!\*)\s*\n(?::PROPERTIES:\s*\n((?::[^:\n]+:[^\n]*\n)*):END:\s*\n)?(.*?)(?=^\*\*(?!\*)|\Z)"
# Capture group 1: header content (can contain ID in v1.6)
# Capture group 2: properties text
# Capture group 3: post content
post_pattern = r"^\*\*(?!\*)([^\n]*)\n(?::PROPERTIES:\s*\n((?::[^:\n]+:[^\n]*\n)*):END:\s*\n)?(.*?)(?=^\*\*(?!\*)|\Z)"
post_matches = re.finditer(
post_pattern, posts_content, re.DOTALL | re.MULTILINE
)
for post_match in post_matches:
properties_text = post_match.group(1) or ""
content_text = post_match.group(2).strip() if post_match.group(2) else ""
header_text = post_match.group(1).strip() if post_match.group(1) else ""
properties_text = post_match.group(2) or ""
content_text = post_match.group(3).strip() if post_match.group(3) else ""
post: Dict[str, Any] = {
"id": "",
@ -124,6 +399,17 @@ def parse_org_social(url: str) -> Dict[str, Any]:
"poll_options": [],
}
# First check if ID is in header (v1.6 feature)
# Header ID takes priority over property drawer ID
if header_text:
# RFC 3339 format: ####-##-##T##:##:##[+-]####
header_id_match = re.match(
r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{2}:?\d{2}$",
header_text,
)
if header_id_match:
post["id"] = header_text
# Parse properties
if properties_text:
# Use [ \t]* instead of \s* to avoid capturing newlines
@ -134,7 +420,8 @@ def parse_org_social(url: str) -> Dict[str, Any]:
# Only add non-empty properties
if prop_value:
post["properties"][prop_name] = prop_value
if prop_name == "id":
# If ID not already set from header, use property ID
if prop_name == "id" and not post["id"]:
post["id"] = prop_value
# Extract mentions from content
@ -174,6 +461,10 @@ def parse_org_social_content(content: str) -> Dict[str, Any]:
"nick": "",
"description": "",
"avatar": "",
"location": "",
"birthday": "",
"language": "",
"pinned": "",
"links": [],
"follows": [],
"contacts": [],
@ -202,6 +493,25 @@ def parse_org_social_content(content: str) -> Dict[str, Any]:
avatar_match = re.search(r"^\s*\#\+AVATAR:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["avatar"] = avatar_match.group(1).strip() if avatar_match else ""
# Parse new v1.6 fields
location_match = re.search(r"^\s*\#\+LOCATION:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["location"] = (
location_match.group(1).strip() if location_match else ""
)
birthday_match = re.search(r"^\s*\#\+BIRTHDAY:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["birthday"] = (
_sanitize_birthday(birthday_match.group(1)) if birthday_match else ""
)
language_match = re.search(r"^\s*\#\+LANGUAGE:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["language"] = (
language_match.group(1).strip() if language_match else ""
)
pinned_match = re.search(r"^\s*\#\+PINNED:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["pinned"] = pinned_match.group(1).strip() if pinned_match else ""
# Parse multiple values
result["metadata"]["links"] = [
match.group(1).strip()
@ -233,14 +543,18 @@ def parse_org_social_content(content: str) -> Dict[str, Any]:
# Split posts by ** headers (exactly 2 asterisks, not 3+)
# Use negative lookahead (?!\*) to ensure we don't match *** or ****
# Use ^ anchor to match ** only at start of line
post_pattern = r"\*\*(?!\*)\s*\n(?::PROPERTIES:\s*\n((?::[^:\n]+:[^\n]*\n)*):END:\s*\n)?(.*?)(?=^\*\*(?!\*)|\Z)"
# Capture group 1: header content (can contain ID in v1.6)
# Capture group 2: properties text
# Capture group 3: post content
post_pattern = r"^\*\*(?!\*)([^\n]*)\n(?::PROPERTIES:\s*\n((?::[^:\n]+:[^\n]*\n)*):END:\s*\n)?(.*?)(?=^\*\*(?!\*)|\Z)"
post_matches = re.finditer(
post_pattern, posts_content, re.DOTALL | re.MULTILINE
)
for post_match in post_matches:
properties_text = post_match.group(1) or ""
content_text = post_match.group(2).strip() if post_match.group(2) else ""
header_text = post_match.group(1).strip() if post_match.group(1) else ""
properties_text = post_match.group(2) or ""
content_text = post_match.group(3).strip() if post_match.group(3) else ""
post: Dict[str, Any] = {
"id": "",
@ -250,6 +564,17 @@ def parse_org_social_content(content: str) -> Dict[str, Any]:
"poll_options": [],
}
# First check if ID is in header (v1.6 feature)
# Header ID takes priority over property drawer ID
if header_text:
# RFC 3339 format: ####-##-##T##:##:##[+-]####
header_id_match = re.match(
r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{2}:?\d{2}$",
header_text,
)
if header_id_match:
post["id"] = header_text
# Parse properties
if properties_text:
# Use [ \t]* instead of \s* to avoid capturing newlines
@ -260,7 +585,8 @@ def parse_org_social_content(content: str) -> Dict[str, Any]:
# Only add non-empty properties
if prop_value:
post["properties"][prop_name] = prop_value
if prop_name == "id":
# If ID not already set from header, use property ID
if prop_name == "id" and not post["id"]:
post["id"] = prop_value
# Extract mentions from content
@ -295,14 +621,23 @@ def validate_org_social_feed(url: str) -> Tuple[bool, str]:
"""
try:
# Check if URL responds with 200
response = requests.get(url, timeout=5)
response = requests.get(url, timeout=FEED_FETCH_TIMEOUT)
if response.status_code != 200:
return False, f"URL returned status code {response.status_code}"
# Check if URL was redirected
final_url = response.url
if final_url != url and response.history:
logger.info(f"Validation: Redirect detected: {url} -> {final_url}")
_handle_feed_redirect(url, final_url)
# Use final URL for validation
url = final_url
# Update last_successful_fetch since we got a 200 response
_update_feed_last_successful_fetch(url)
content = response.text
# Decode content as UTF-8 explicitly to avoid encoding issues
content = response.content.decode("utf-8")
# Check if content has basic Org Social structure
# At minimum should have at least one #+TITLE, #+NICK, or #+DESCRIPTION (case insensitive)

524
app/feeds/tasks.py
View file

@ -15,181 +15,126 @@ def discover_feeds_from_relay_nodes():
Periodic task to discover new feeds from other Org Social Relay nodes.
This task:
1. Fetches the list of relay nodes from the public URL
1. Reads the list of relay nodes from a local file
2. Filters out our own domain to avoid self-discovery
3. Calls each relay node's /feeds endpoint to get their registered feeds
4. Stores newly discovered feeds in our local database
"""
import django
from pathlib import Path
django.setup()
from django.conf import settings
from app.bridge.models import is_bridge_feed_url
from .models import Feed
from .parser import validate_org_social_feed
# URLs to fetch feeds from
feed_sources = [
{
"name": "relay nodes",
"url": "https://cdn.jsdelivr.net/gh/tanrax/org-social/org-social-relay-list.txt",
"type": "relay_nodes",
},
{
"name": "public register",
"url": "https://raw.githubusercontent.com/tanrax/org-social/main/registers.txt",
"type": "direct_feeds",
},
]
# Get the project root directory (where manage.py is located)
project_root = Path(__file__).resolve().parent.parent.parent
relay_list_path = project_root / "relay-list.txt"
total_discovered = 0
for source in feed_sources:
logger.info(f"Fetching feeds from {source['name']}: {source['url']}")
logger.info(f"Reading relay nodes from: {relay_list_path}")
try:
# Fetch the list
response = requests.get(source["url"], timeout=5)
response.raise_for_status()
try:
# Read the local file
with open(relay_list_path, "r", encoding="utf-8") as f:
content = f.read()
# The file might be empty or contain one URL per line
urls = [line.strip() for line in response.text.split("\n") if line.strip()]
# The file might be empty or contain one URL per line
relay_nodes = [line.strip() for line in content.split("\n") if line.strip()]
if source["type"] == "direct_feeds":
# For direct feeds (registers.txt), validate and add them directly
logger.info(f"Found {len(urls)} direct feeds to validate")
# Filter out our own domain to avoid self-discovery
site_domain = settings.SITE_DOMAIN
filtered_nodes = []
for node_url in relay_nodes:
# Normalize the URL for comparison
normalized_node = (
node_url.replace("http://", "").replace("https://", "").strip("/")
)
normalized_site = site_domain.strip("/")
for feed_url in urls:
if not feed_url.strip():
continue
if normalized_node != normalized_site:
filtered_nodes.append(node_url)
else:
logger.info(f"Skipping own domain: {node_url}")
feed_url = feed_url.strip()
relay_nodes = filtered_nodes
# Check if we already have this feed
if Feed.objects.filter(url=feed_url).exists():
continue
if not relay_nodes:
logger.info("No relay nodes found in the list after filtering own domain")
return
# Validate the feed before adding it
logger.info(f"Validating direct feed: {feed_url}")
is_valid, error_message = validate_org_social_feed(feed_url)
logger.info(
f"Found {len(relay_nodes)} relay nodes to check (excluding own domain)"
)
if not is_valid:
logger.warning(
f"Skipping invalid direct feed {feed_url}: {error_message}"
)
continue
for node_url in relay_nodes:
try:
# Ensure the URL has proper format
if not node_url.startswith(("http://", "https://")):
node_url = f"http://{node_url}"
# Create the feed
try:
Feed.objects.create(url=feed_url)
total_discovered += 1
logger.info(f"Added direct feed: {feed_url}")
except Exception as e:
logger.error(f"Failed to create direct feed {feed_url}: {e}")
# Call the /feeds endpoint on each relay node
feeds_url = f"{node_url}/feeds"
feeds_response = requests.get(feeds_url, timeout=10)
feeds_response.raise_for_status()
elif source["type"] == "relay_nodes":
# For relay nodes, get their feeds endpoints
relay_nodes = urls
feeds_data = feeds_response.json()
# Filter out our own domain to avoid self-discovery
site_domain = settings.SITE_DOMAIN
filtered_nodes = []
for node_url in relay_nodes:
# Normalize the URL for comparison
normalized_node = (
node_url.replace("http://", "")
.replace("https://", "")
.strip("/")
)
normalized_site = site_domain.strip("/")
# Check if response has expected format
if feeds_data.get("type") == "Success" and "data" in feeds_data:
feeds_list = feeds_data["data"]
if normalized_node != normalized_site:
filtered_nodes.append(node_url)
else:
logger.info(f"Skipping own domain: {node_url}")
for feed_url in feeds_list:
if isinstance(feed_url, str) and feed_url.strip():
feed_url = feed_url.strip()
relay_nodes = filtered_nodes
# Bridge virtual feeds are not real accounts
if is_bridge_feed_url(feed_url):
continue
if not relay_nodes:
logger.info(
"No relay nodes found in the list after filtering own domain"
)
continue
# Check if we already have this feed
if Feed.objects.filter(url=feed_url).exists():
continue
logger.info(
f"Found {len(relay_nodes)} relay nodes to check (excluding own domain)"
)
# Validate the feed before adding it
logger.info(f"Validating discovered feed: {feed_url}")
is_valid, error_message = validate_org_social_feed(feed_url)
for node_url in relay_nodes:
try:
# Ensure the URL has proper format
if not node_url.startswith(("http://", "https://")):
node_url = f"http://{node_url}"
if not is_valid:
logger.warning(
f"Skipping invalid feed {feed_url}: {error_message}"
)
continue
# Call the /feeds endpoint on each relay node
feeds_url = f"{node_url}/feeds"
feeds_response = requests.get(feeds_url, timeout=10)
feeds_response.raise_for_status()
# Create the feed
try:
Feed.objects.create(url=feed_url)
total_discovered += 1
logger.info(
f"Discovered and validated new feed: {feed_url}"
)
except Exception as e:
logger.error(f"Failed to create feed {feed_url}: {e}")
feeds_data = feeds_response.json()
logger.info(f"Successfully checked relay node: {node_url}")
# Check if response has expected format
if feeds_data.get("type") == "Success" and "data" in feeds_data:
feeds_list = feeds_data["data"]
except requests.RequestException as e:
logger.warning(f"Failed to fetch feeds from relay node {node_url}: {e}")
except ValueError as e:
logger.warning(f"Invalid JSON response from relay node {node_url}: {e}")
except Exception as e:
logger.error(f"Unexpected error checking relay node {node_url}: {e}")
for feed_url in feeds_list:
if isinstance(feed_url, str) and feed_url.strip():
feed_url = feed_url.strip()
# Check if we already have this feed
if Feed.objects.filter(url=feed_url).exists():
continue
# Validate the feed before adding it
logger.info(
f"Validating discovered feed: {feed_url}"
)
is_valid, error_message = validate_org_social_feed(
feed_url
)
if not is_valid:
logger.warning(
f"Skipping invalid feed {feed_url}: {error_message}"
)
continue
# Create the feed
try:
Feed.objects.create(url=feed_url)
total_discovered += 1
logger.info(
f"Discovered and validated new feed: {feed_url}"
)
except Exception as e:
logger.error(
f"Failed to create feed {feed_url}: {e}"
)
logger.info(f"Successfully checked relay node: {node_url}")
except requests.RequestException as e:
logger.warning(
f"Failed to fetch feeds from relay node {node_url}: {e}"
)
except ValueError as e:
logger.warning(
f"Invalid JSON response from relay node {node_url}: {e}"
)
except Exception as e:
logger.error(
f"Unexpected error checking relay node {node_url}: {e}"
)
except requests.RequestException as e:
logger.error(f"Failed to fetch {source['name']} from {source['url']}: {e}")
except Exception as e:
logger.error(f"Unexpected error processing {source['name']}: {e}")
except FileNotFoundError as e:
logger.error(f"Relay list file not found: {relay_list_path}: {e}")
except IOError as e:
logger.error(f"Failed to read relay list from {relay_list_path}: {e}")
except Exception as e:
logger.error(f"Unexpected error processing relay list: {e}")
logger.info(
f"Feed discovery completed. Total new feeds discovered: {total_discovered}"
@ -211,6 +156,7 @@ def discover_new_feeds_from_follows():
django.setup()
from app.bridge.models import is_bridge_feed_url
from .models import Feed, Profile, Follow
from .parser import parse_org_social, validate_org_social_feed
@ -232,10 +178,13 @@ def discover_new_feeds_from_follows():
for feed in all_feeds:
try:
# Parse the org social file
# Parse the org social file (may update feed URL if redirected)
parsed_data = parse_org_social(feed.url)
successful_parses += 1
# Refresh feed from database (URL may have changed due to redirect)
feed.refresh_from_db()
# Extract follow URLs from metadata
follows = parsed_data.get("metadata", {}).get("follows", [])
@ -258,6 +207,10 @@ def discover_new_feeds_from_follows():
"nick": metadata.get("nick", ""),
"description": metadata.get("description", ""),
"avatar": metadata.get("avatar", ""),
"location": metadata.get("location", ""),
"birthday": metadata.get("birthday") or None,
"language": metadata.get("language", ""),
"pinned": metadata.get("pinned", ""),
"version": content_hash,
},
)
@ -276,7 +229,10 @@ def discover_new_feeds_from_follows():
# Check if feed already exists
existing_feed = Feed.objects.filter(url=follow_url).first()
if not existing_feed:
# Bridge virtual feeds help users connect to external
# content; the follow relationship is kept below, but the
# bridge is never registered as a real feed
if not existing_feed and not is_bridge_feed_url(follow_url):
# Validate the feed before adding it
logger.info(f"Validating discovered follow feed: {follow_url}")
is_valid, error_message = validate_org_social_feed(follow_url)
@ -403,10 +359,13 @@ def scan_feeds():
for feed in all_feeds:
try:
# Parse the org social file
# Parse the org social file (may update feed URL if redirected)
parsed_data = parse_org_social(feed.url)
successful_scans += 1
# Refresh feed from database (URL may have changed due to redirect)
feed.refresh_from_db()
metadata = parsed_data.get("metadata", {})
posts_data = parsed_data.get("posts", [])
@ -422,6 +381,10 @@ def scan_feeds():
"nick": metadata.get("nick", ""),
"description": metadata.get("description", ""),
"avatar": metadata.get("avatar", ""),
"location": metadata.get("location", ""),
"birthday": metadata.get("birthday") or None,
"language": metadata.get("language", ""),
"pinned": metadata.get("pinned", ""),
"version": content_hash,
},
)
@ -437,13 +400,14 @@ def scan_feeds():
profile.nick = metadata.get("nick", "")
profile.description = metadata.get("description", "")
profile.avatar = metadata.get("avatar", "")
profile.location = metadata.get("location", "")
profile.birthday = metadata.get("birthday") or None
profile.language = metadata.get("language", "")
profile.pinned = metadata.get("pinned", "")
profile.version = content_hash
profile.save()
profiles_updated += 1
logger.info(f"Updated profile: {profile.nick} ({feed.url})")
else:
# No changes detected, skip processing
continue
# Update profile relationships (clear and recreate)
profile.links.all().delete()
@ -476,6 +440,16 @@ def scan_feeds():
content = post_data.get("content", "")
properties = post_data.get("properties", {})
# Parse post_id as timestamp for created_at
# post_id is in RFC 3339 format (e.g., "2025-01-01T12:00:00+00:00")
post_created_at = timezone.now() # Default fallback
try:
post_created_at = date_parser.parse(post_id)
except Exception as e:
logger.warning(
f"Failed to parse post_id {post_id} as timestamp: {e}. Using current time."
)
# Extract group name from GROUP property
# Format: "Emacs https://org-social-relay.andros.dev" or just "Emacs"
# Group names can have spaces and capitals - we slugify them
@ -510,14 +484,67 @@ def scan_feeds():
"reply_to": properties.get("reply_to", ""),
"mood": properties.get("mood", ""),
"group": group_slug,
"include": properties.get("include", ""),
"poll_end": None,
"created_at": post_created_at,
},
)
if post_created:
posts_created += 1
# Queue outgoing webmentions for external links in the post
from .webmentions import queue_webmentions_for_post
try:
queue_webmentions_for_post(profile, post_id, content)
except Exception as e:
logger.warning(
f"Failed to queue webmentions for post {post_id}: {e}"
)
# Publish notifications for NEW posts
from .notification_publisher import publish_notification
# Check if this is a reply (with or without mood/reaction)
if post.reply_to:
reply_to_parts = post.reply_to.split("#")
if len(reply_to_parts) == 2:
replied_feed_url = reply_to_parts[0]
# Determine if it's a reaction or a reply
if post.mood and post.mood.strip():
# It's a reaction
publish_notification(
target_feed_url=replied_feed_url,
notification_type="reaction",
post_url=f"{feed.url}#{post_id}",
emoji=post.mood,
parent=post.reply_to,
)
else:
# It's a regular reply
publish_notification(
target_feed_url=replied_feed_url,
notification_type="reply",
post_url=f"{feed.url}#{post_id}",
parent=post.reply_to,
)
# Check if this is a boost
if post.include:
include_parts = post.include.split("#")
if len(include_parts) == 2:
boosted_feed_url = include_parts[0]
publish_notification(
target_feed_url=boosted_feed_url,
notification_type="boost",
post_url=f"{feed.url}#{post_id}",
boosted=post.include,
)
else:
# Update existing post
content_changed = post.content != content
post.content = content
post.language = properties.get("lang", "")
post.tags = properties.get("tags", "")
@ -525,9 +552,22 @@ def scan_feeds():
post.reply_to = properties.get("reply_to", "")
post.mood = properties.get("mood", "")
post.group = group_slug
post.include = properties.get("include", "")
post.save()
posts_updated += 1
# Links added by an edit get their webmention queued too;
# already-known (source, target) pairs are never re-sent
if content_changed:
from .webmentions import queue_webmentions_for_post
try:
queue_webmentions_for_post(profile, post_id, content)
except Exception as e:
logger.warning(
f"Failed to queue webmentions for post {post_id}: {e}"
)
# Handle poll_end if present
poll_end_str = properties.get("poll_end", "")
if poll_end_str:
@ -584,13 +624,15 @@ def scan_feeds():
f"Failed to process poll vote for post {post_id}: {e}"
)
# Handle mentions
# Handle mentions - FIXED to detect new mentions
mentions_data = post_data.get("mentions", [])
if mentions_data:
# Clear existing mentions for this post
post.mentions.all().delete()
# Get existing mentions for this post
existing_mentions = set(
post.mentions.values_list("mentioned_profile__feed", flat=True)
)
# Create new mentions
# Process each mention
for mention_info in mentions_data:
mention_url = mention_info.get("url", "").strip()
mention_nickname = mention_info.get("nickname", "").strip()
@ -607,16 +649,30 @@ def scan_feeds():
feed=base_mention_url
)
# Create the mention
Mention.objects.get_or_create(
post=post,
mentioned_profile=mentioned_profile,
defaults={"nickname": mention_nickname},
)
# Only create if it doesn't exist (to detect new mentions)
if base_mention_url not in existing_mentions:
mention, mention_created = (
Mention.objects.get_or_create(
post=post,
mentioned_profile=mentioned_profile,
defaults={"nickname": mention_nickname},
)
)
# If this is a NEW mention, publish notification
if mention_created:
from .notification_publisher import (
publish_notification,
)
publish_notification(
target_feed_url=base_mention_url,
notification_type="mention",
post_url=f"{feed.url}#{post_id}",
)
except Profile.DoesNotExist:
# The mentioned profile doesn't exist in our database
# We could optionally log this or create a placeholder profile
logger.debug(f"Mentioned profile not found: {mention_url}")
continue
except Exception as e:
@ -624,6 +680,27 @@ def scan_feeds():
f"Failed to create mention for {mention_url} in post {post_id}: {e}"
)
# Detect and remove deleted posts
# Get all post IDs from the current feed scan
current_post_ids = {post_data.get("id", "") for post_data in posts_data}
current_post_ids.discard("") # Remove empty IDs
# Get all post IDs currently in database for this profile
existing_posts = Post.objects.filter(profile=profile)
existing_post_ids = set(existing_posts.values_list("post_id", flat=True))
# Find posts that are in DB but not in current feed (deleted posts)
deleted_post_ids = existing_post_ids - current_post_ids
if deleted_post_ids:
# Delete posts that no longer exist in the feed
deleted_count = Post.objects.filter(
profile=profile, post_id__in=deleted_post_ids
).delete()[0]
logger.info(
f"Removed {deleted_count} deleted post(s) from {feed.url}: {deleted_post_ids}"
)
except requests.RequestException as e:
failed_scans += 1
logger.warning(f"Failed to fetch/parse feed {feed.url}: {e}")
@ -642,15 +719,148 @@ def scan_feeds():
f"Posts updated: {posts_updated}"
)
# Update global relay metadata BEFORE clearing cache
# This ensures the new ETag/Last-Modified are ready when cache is cleared
from .models import RelayMetadata
RelayMetadata.update_global_metadata()
logger.info("Updated global relay metadata (ETag and Last-Modified)")
# Clear cache AFTER scanning to ensure next requests get fresh data
# This way during scan users see complete old cached data (consistent),
# and after scan they see complete new data (also consistent)
from django.core.cache import cache
# Invalidate middleware cache for headers (will be recreated from DB on next request)
cache.delete("relay_metadata_headers")
# Clear all endpoint caches
cache.clear()
logger.info("Cache cleared after feed scanning - next requests will get fresh data")
WEBMENTION_MAX_ATTEMPTS = 5
WEBMENTION_BATCH_SIZE = 50
def _send_pending_webmentions_impl():
"""
Implementation of the webmention delivery logic, separated from the
periodic task to allow for easier testing.
Returns:
dict: Counters of sent / no_endpoint / failed webmentions
"""
from .models import OutgoingWebmention
from .webmentions import (
discover_webmention_endpoint,
is_safe_endpoint,
send_webmention,
)
now = timezone.now()
candidates = OutgoingWebmention.objects.filter(
status__in=[
OutgoingWebmention.STATUS_PENDING,
OutgoingWebmention.STATUS_FAILED,
],
attempts__lt=WEBMENTION_MAX_ATTEMPTS,
).order_by("created_at")[: WEBMENTION_BATCH_SIZE * 2]
counters = {"sent": 0, "no_endpoint": 0, "failed": 0}
processed = 0
for webmention in candidates:
if processed >= WEBMENTION_BATCH_SIZE:
break
# Exponential backoff between retries: 1h, 2h, 4h, 8h...
if webmention.last_attempt_at:
backoff = timedelta(hours=2 ** (webmention.attempts - 1))
if now < webmention.last_attempt_at + backoff:
continue
processed += 1
webmention.last_attempt_at = now
webmention.attempts += 1
try:
endpoint = webmention.endpoint or discover_webmention_endpoint(
webmention.target
)
if endpoint is None:
# Permanent: the target does not support webmentions
webmention.status = OutgoingWebmention.STATUS_NO_ENDPOINT
counters["no_endpoint"] += 1
webmention.save()
continue
if not is_safe_endpoint(endpoint):
logger.warning(
f"Rejecting unsafe webmention endpoint {endpoint} "
f"for target {webmention.target}"
)
webmention.status = OutgoingWebmention.STATUS_NO_ENDPOINT
counters["no_endpoint"] += 1
webmention.save()
continue
webmention.endpoint = endpoint
status_code = send_webmention(
endpoint, webmention.source, webmention.target
)
webmention.response_code = status_code
if 200 <= status_code < 300:
webmention.status = OutgoingWebmention.STATUS_SENT
counters["sent"] += 1
logger.info(
f"Webmention sent: {webmention.source} -> {webmention.target} "
f"({status_code})"
)
else:
webmention.status = OutgoingWebmention.STATUS_FAILED
counters["failed"] += 1
logger.warning(
f"Webmention rejected by {endpoint} with HTTP {status_code} "
f"({webmention.source} -> {webmention.target})"
)
except requests.RequestException as e:
webmention.status = OutgoingWebmention.STATUS_FAILED
counters["failed"] += 1
logger.warning(f"Webmention delivery error for {webmention.target}: {e}")
webmention.save()
if processed:
logger.info(
f"Webmention delivery completed. "
f"Sent: {counters['sent']}, "
f"No endpoint: {counters['no_endpoint']}, "
f"Failed: {counters['failed']}"
)
return counters
@periodic_task(crontab(minute="*/5")) # Run every 5 minutes
def send_pending_webmentions():
"""
Periodic task to deliver queued outgoing webmentions.
For each pending (or retriable failed) webmention this task discovers
the target's webmention endpoint and POSTs source/target to it, as
described in https://www.w3.org/TR/webmention/. Targets without an
endpoint are marked permanently so they are never fetched again.
"""
import django
django.setup()
return _send_pending_webmentions_impl()
def _cleanup_stale_feeds_impl():
"""
Implementation of stale feed cleanup logic.

231
app/feeds/test_discover_relay_nodes.py Normal file
View file

@ -0,0 +1,231 @@
from django.test import TestCase
from django.conf import settings
from unittest.mock import patch, Mock
from app.feeds.models import Feed
from app.feeds.tasks import discover_feeds_from_relay_nodes
class DiscoverRelayNodesTest(TestCase):
"""Test cases for discover_feeds_from_relay_nodes task using Given/When/Then structure."""
def setUp(self):
"""Set up test fixtures."""
# Clear any existing feeds
Feed.objects.all().delete()
def test_discovers_feeds_from_real_relay_list(self):
"""Test that feeds are discovered from the actual relay list file."""
# Given: The real relay list file exists (data/relay-list.txt)
# And we mock the HTTP responses from relays
# Mock relay response
mock_relay_response = Mock()
mock_relay_response.status_code = 200
mock_relay_response.json.return_value = {
"type": "Success",
"data": [
"https://test-feed1.com/social.org",
"https://test-feed2.com/social.org",
],
}
mock_relay_response.raise_for_status = Mock()
# Mock feed validation responses
mock_feed_response = Mock()
mock_feed_response.status_code = 200
mock_feed_response.text = """#+TITLE: Test Feed
#+NICK: testuser
#+DESCRIPTION: Test description
* Posts
"""
mock_feed_response.content = mock_feed_response.text.encode("utf-8")
mock_feed_response.history = []
def mock_requests_get(url, timeout=None):
if "/feeds" in url:
return mock_relay_response
# Feed validation requests
mock_feed_response.url = url
mock_feed_response.raise_for_status = Mock()
return mock_feed_response
# When: We run the discovery task
with patch("requests.get", side_effect=mock_requests_get):
discover_feeds_from_relay_nodes()
# Then: Feeds should be discovered (the real relay list has relays)
# We can't predict exact count since it depends on real file content
# So we just verify the function runs without errors
self.assertGreaterEqual(Feed.objects.count(), 0)
def test_handles_relay_connection_failure(self):
"""Test that the task handles relay connection failures gracefully."""
# Given: The real relay list file exists
# And relay connections will fail
# Mock a connection error
with patch("requests.get", side_effect=Exception("Connection failed")):
# When: We run the discovery task
initial_count = Feed.objects.count()
discover_feeds_from_relay_nodes()
# Then: No new feeds should be created and no exception raised
self.assertEqual(Feed.objects.count(), initial_count)
def test_skips_existing_feeds(self):
"""Test that the task skips feeds that already exist."""
# Given: A feed already exists
Feed.objects.create(url="https://existing-feed.com/social.org")
initial_count = Feed.objects.count()
# Mock relay response with the existing feed
mock_relay_response = Mock()
mock_relay_response.status_code = 200
mock_relay_response.json.return_value = {
"type": "Success",
"data": [
"https://existing-feed.com/social.org", # Already exists
],
}
mock_relay_response.raise_for_status = Mock()
def mock_requests_get(url, timeout=None):
if "/feeds" in url:
return mock_relay_response
# Should not validate existing feed
if "existing-feed" in url:
self.fail("Should not validate existing feed")
return Mock()
# When: We run the discovery task
with patch("requests.get", side_effect=mock_requests_get):
discover_feeds_from_relay_nodes()
# Then: No new feeds should be added
self.assertEqual(Feed.objects.count(), initial_count)
def test_handles_invalid_feed_during_validation(self):
"""Test that the task skips invalid feeds during validation."""
# Given: The real relay list file exists
# Mock relay response with valid and invalid feeds
mock_relay_response = Mock()
mock_relay_response.status_code = 200
mock_relay_response.json.return_value = {
"type": "Success",
"data": [
"https://valid-feed.com/social.org",
"https://invalid-feed.com/social.org",
],
}
mock_relay_response.raise_for_status = Mock()
# Mock feed validation responses
def mock_requests_get(url, timeout=None):
if "/feeds" in url:
return mock_relay_response
# Valid feed response
if "valid-feed" in url:
mock_response = Mock()
mock_response.status_code = 200
mock_response.text = """#+TITLE: Valid Feed
#+NICK: validuser
#+DESCRIPTION: Valid description
* Posts
"""
mock_response.content = mock_response.text.encode("utf-8")
mock_response.url = url
mock_response.history = []
mock_response.raise_for_status = Mock()
return mock_response
# Invalid feed response (missing required fields)
if "invalid-feed" in url:
mock_response = Mock()
mock_response.status_code = 200
mock_response.text = "Invalid content without Org Social headers"
mock_response.content = mock_response.text.encode("utf-8")
mock_response.url = url
mock_response.history = []
mock_response.raise_for_status = Mock()
return mock_response
# Default response for any other URL
mock_response = Mock()
mock_response.status_code = 404
mock_response.raise_for_status = Mock(side_effect=Exception("404"))
return mock_response
# When: We run the discovery task
initial_count = Feed.objects.count()
with patch("requests.get", side_effect=mock_requests_get):
discover_feeds_from_relay_nodes()
# Then: Only the valid feed should be added (or none if validation failed for both)
# We verify that invalid feed was not added
self.assertFalse(
Feed.objects.filter(url="https://invalid-feed.com/social.org").exists()
)
# And if valid feed was processed, it should exist
# (we're more lenient here as the test DB might interfere)
if Feed.objects.count() > initial_count:
self.assertTrue(
Feed.objects.filter(url="https://valid-feed.com/social.org").exists()
)
def test_filters_out_own_domain(self):
"""Test that the task filters out its own domain from relay list."""
# Given: The real relay list file exists
# We verify the function doesn't try to fetch from its own domain
own_domain = settings.SITE_DOMAIN
def mock_requests_get(url, timeout=None):
# Should not be called for our own domain
if own_domain in url and "/feeds" in url:
self.fail(f"Should not fetch from own domain: {url}")
# Mock response for other relays
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {"type": "Success", "data": []}
mock_response.raise_for_status = Mock()
return mock_response
# When: We run the discovery task
with patch("requests.get", side_effect=mock_requests_get):
discover_feeds_from_relay_nodes()
# Then: The function should complete without calling own domain
# (if it called own domain, the test would fail in mock_requests_get)
def test_skips_bridge_feeds_from_relay_nodes(self):
"""Test that bridge virtual feeds listed by other relays are skipped."""
# Given: A relay node listing only bridge virtual feeds
mock_relay_response = Mock()
mock_relay_response.status_code = 200
mock_relay_response.json.return_value = {
"type": "Success",
"data": [
"https://other-relay.org/bridge/rss/"
"?url=https%3A%2F%2Frss.arxiv.org%2Frss%2Fquant-ph",
"https://other-relay.org/bridge/activitypub/@user@instance.tld/",
],
}
mock_relay_response.raise_for_status = Mock()
def mock_requests_get(url, timeout=None):
if "/feeds" in url:
return mock_relay_response
# Bridge feeds must not even be validated
self.fail(f"Should not validate bridge feed: {url}")
# When: We run the discovery task
with patch("requests.get", side_effect=mock_requests_get):
discover_feeds_from_relay_nodes()
# Then: No bridge feed is registered
self.assertEqual(Feed.objects.count(), 0)

6
app/feeds/test_feed_cleanup.py
View file

@ -25,6 +25,9 @@ class FeedCleanupTest(TestCase):
* Posts
"""
mock_response.content = mock_response.text.encode("utf-8")
mock_response.url = feed_url # No redirect
mock_response.history = [] # No redirect history
mock_response.raise_for_status = Mock()
# When: We parse the feed and it returns HTTP 200
@ -55,6 +58,9 @@ class FeedCleanupTest(TestCase):
* Posts
"""
mock_response.content = mock_response.text.encode("utf-8")
mock_response.url = feed_url # No redirect
mock_response.history = [] # No redirect history
# When: We validate the feed and it returns HTTP 200
with patch("requests.get", return_value=mock_response):

104
app/feeds/test_feeds.py
View file

@ -2,7 +2,7 @@ import json
from django.test import TestCase
from rest_framework.test import APIClient
from rest_framework import status
from .models import Feed
from app.feeds.models import Feed
class FeedsViewTest(TestCase):
@ -277,5 +277,105 @@ class FeedsViewTest(TestCase):
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(response.data["type"], "Error")
self.assertIn("Invalid Org Social feed", response.data["errors"][0])
self.assertIn("missing basic metadata", response.data["errors"][0])
# Accept either encoding errors or missing metadata errors
error_message = response.data["errors"][0]
self.assertTrue(
"missing basic metadata" in error_message
or "Validation error" in error_message,
f"Expected validation error, got: {error_message}",
)
self.assertIsNone(response.data["data"])
def test_get_feeds_has_caching_headers(self):
"""Test GET /feeds returns ETag and Last-Modified headers."""
# Given: Some feeds in the database
Feed.objects.create(url="https://example.com/social.org")
Feed.objects.create(url="https://test.com/social.org")
# When: We request the feeds list
response = self.client.get(self.feeds_url)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
# Then: ETag should be properly formatted (quoted)
etag = response["ETag"]
self.assertTrue(etag.startswith('"') and etag.endswith('"'))
# Then: Last-Modified should be in RFC 2822 format
last_modified = response["Last-Modified"]
self.assertIsNotNone(last_modified)
# Format example: "Wed, 05 Nov 2025 10:15:00 GMT"
self.assertIn("GMT", last_modified)
def test_get_feeds_etag_is_global_and_consistent(self):
"""Test GET /feeds ETag is global and doesn't change when feeds are added."""
# Given: Initial feeds
Feed.objects.create(url="https://example.com/social.org")
# When: We request the feeds list
response1 = self.client.get(self.feeds_url)
etag1 = response1["ETag"]
# When: We add a new feed
Feed.objects.create(url="https://new.com/social.org")
# When: We request the feeds list again
response2 = self.client.get(self.feeds_url)
etag2 = response2["ETag"]
# Then: ETag should be the same (global ETag, only changes with scan_feeds)
self.assertEqual(etag1, etag2)
def test_get_empty_feeds_has_caching_headers(self):
"""Test GET /feeds returns caching headers even when empty."""
# Given: No feeds in the database
Feed.objects.all().delete()
# When: We request the feeds list
response = self.client.get(self.feeds_url)
# Then: Should still have caching headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_get_feeds_excludes_bridge_feeds(self):
"""Test GET /feeds never lists bridge virtual feeds."""
# Given: A real feed and two bridge virtual feeds in the database
Feed.objects.create(url="https://example.com/social.org")
Feed.objects.create(
url="https://relay.org-social.org/bridge/rss/"
"?url=https%3A%2F%2Frss.arxiv.org%2Frss%2Fquant-ph"
)
Feed.objects.create(
url="https://relay.org-social.org/bridge/activitypub/@user@instance.tld/"
)
# When: We request the feeds list
response = self.client.get(self.feeds_url)
# Then: Only the real feed is listed
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["data"], ["https://example.com/social.org"])
def test_post_bridge_feed_is_rejected(self):
"""Test POST /feeds rejects bridge virtual feeds."""
# Given: The URL of a bridge virtual feed
bridge_url = (
"https://relay.org-social.org/bridge/rss/"
"?url=https%3A%2F%2Fxkcd.com%2Frss.xml"
)
# When: We try to register it
response = self.client.post(
self.feeds_url, {"feed": bridge_url}, format="json"
)
# Then: The request is rejected and no feed is created
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(response.data["type"], "Error")
self.assertIn(
"Bridge virtual feeds cannot be registered", response.data["errors"]
)
self.assertFalse(Feed.objects.filter(url=bridge_url).exists())

12
app/feeds/test_mentions.py
View file

@ -1,7 +1,7 @@
from django.test import TestCase
from rest_framework.test import APIClient
from rest_framework import status
from .models import Profile, Post, Mention
from app.feeds.models import Profile, Post, Mention
class MentionsViewTest(TestCase):
@ -88,10 +88,13 @@ class MentionsViewTest(TestCase):
meta = response.data["meta"]
self.assertIn("feed", meta)
self.assertIn("total", meta)
self.assertIn("version", meta)
self.assertEqual(meta["feed"], "https://alice.example.com/social.org")
self.assertEqual(meta["total"], 2)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_get_mentions_missing_feed_parameter(self):
"""Test GET /mentions returns error when feed parameter is missing."""
# Given: A request without feed parameter
@ -190,7 +193,10 @@ class MentionsViewTest(TestCase):
meta = response.data["meta"]
self.assertIn("feed", meta)
self.assertIn("total", meta)
self.assertIn("version", meta)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_mentions_view_only_get_allowed(self):
"""Test that only GET method is allowed on mentions endpoint."""

482
app/feeds/test_parser.py
View file

@ -1,5 +1,6 @@
import os
import tempfile
from unittest.mock import Mock, patch
from django.test import TestCase
@ -486,3 +487,484 @@ Test content
f"Property '{key}' has value starting with colon: '{value}'. "
f"This indicates the regex is capturing the next property.",
)
def test_parse_emoji_with_skin_tone_utf8(self):
"""Test parsing emojis with skin tone modifiers in UTF-8 encoding.
This test validates that emojis with skin tone modifiers (like 🙌🏻)
are correctly parsed and stored in UTF-8 encoding, not double-encoded.
"""
# Given: An org social content with emoji containing skin tone modifier
content = """#+TITLE: Test
#+NICK: test_user
* Posts
**
:PROPERTIES:
:ID: 2025-11-13T12:05:35+0100
:REPLY_TO: https://example.com/social.org#2025-11-13T10:00:00+0100
:MOOD: 🙌🏻
:END:
Great work!
"""
# When: We parse the content
from app.feeds.parser import parse_org_social_content
result = parse_org_social_content(content)
# Then: The emoji should be correctly parsed
self.assertEqual(len(result["posts"]), 1)
post = result["posts"][0]
# Then: The mood should be the emoji with correct UTF-8 encoding
mood = post["properties"]["mood"]
self.assertEqual(mood, "🙌🏻")
# Then: Verify the bytes are correct UTF-8, not double-encoded
# Correct UTF-8: f0 9f 99 8c f0 9f 8f bb (🙌🏻)
# Double-encoded would be: c3 b0 c2 9f c2 99 c2 8c c3 b0 c2 9f c2 8f c2 bb
mood_bytes = mood.encode("utf-8")
self.assertEqual(mood_bytes.hex(), "f09f998cf09f8fbb")
def test_parse_various_emojis_utf8(self):
"""Test parsing various emojis to ensure UTF-8 encoding is preserved."""
# Given: An org social content with multiple different emojis
content = """#+TITLE: Test
#+NICK: test_user
* Posts
**
:PROPERTIES:
:ID: 2025-01-01T10:00:00+0100
:MOOD: 😃
:END:
Happy post!
**
:PROPERTIES:
:ID: 2025-01-01T10:01:00+0100
:MOOD: 🚀
:END:
Launch post!
**
:PROPERTIES:
:ID: 2025-01-01T10:02:00+0100
:MOOD: 💗
:END:
Love post!
**
:PROPERTIES:
:ID: 2025-01-01T10:03:00+0100
:MOOD: 🎉
:END:
Party post!
"""
# When: We parse the content
from app.feeds.parser import parse_org_social_content
result = parse_org_social_content(content)
# Then: All emojis should be correctly parsed
self.assertEqual(len(result["posts"]), 4)
# Then: Verify each emoji and its UTF-8 bytes
expected_emojis = [
("😃", "f09f9883"),
("🚀", "f09f9a80"),
("💗", "f09f9297"),
("🎉", "f09f8e89"),
]
for i, (expected_emoji, expected_bytes) in enumerate(expected_emojis):
post = result["posts"][i]
mood = post["properties"]["mood"]
self.assertEqual(mood, expected_emoji)
self.assertEqual(mood.encode("utf-8").hex(), expected_bytes)
@patch("app.feeds.parser.requests.get")
def test_parse_feed_with_missing_charset_header(self, mock_get):
"""Test parsing a feed when server doesn't specify charset in Content-Type.
This test validates that the parser correctly handles UTF-8 content
even when the server doesn't specify charset in the Content-Type header,
which would cause requests library to default to ISO-8859-1 encoding.
"""
# Given: A feed content with emoji that would be double-encoded if using response.text
content_with_emoji = """#+TITLE: Test
#+NICK: test_user
* Posts
**
:PROPERTIES:
:ID: 2025-11-13T12:05:35+0100
:MOOD: 🙌🏻
:END:
Great work!
"""
# Given: Mock response that simulates server without charset in Content-Type
mock_response = Mock()
mock_response.status_code = 200
mock_response.encoding = "ISO-8859-1" # requests default when no charset
mock_response.content = content_with_emoji.encode("utf-8") # Raw UTF-8 bytes
mock_response.url = "https://example.com/social.org" # No redirect
mock_response.history = [] # No redirect history
mock_response.raise_for_status = Mock()
mock_get.return_value = mock_response
# When: We parse the feed from URL
from app.feeds.parser import parse_org_social
result = parse_org_social("https://example.com/social.org")
# Then: The emoji should be correctly parsed (not double-encoded)
self.assertEqual(len(result["posts"]), 1)
post = result["posts"][0]
mood = post["properties"]["mood"]
self.assertEqual(mood, "🙌🏻")
# Then: Verify the bytes are correct UTF-8, not double-encoded
mood_bytes = mood.encode("utf-8")
self.assertEqual(mood_bytes.hex(), "f09f998cf09f8fbb")
# Then: Verify we're using response.content, not response.text
# This is critical to avoid double-encoding
from app.feeds.parser import FEED_FETCH_TIMEOUT
mock_get.assert_called_once_with(
"https://example.com/social.org", timeout=FEED_FETCH_TIMEOUT
)
def test_parse_v16_metadata_fields(self):
"""Test parsing v1.6 metadata fields (LOCATION, BIRTHDAY, LANGUAGE, PINNED)."""
# Given: An org social content with v1.6 metadata fields
content = """#+TITLE: Test User Profile
#+NICK: test_user
#+DESCRIPTION: Test description
#+AVATAR: https://example.com/avatar.jpg
#+LOCATION: Valencia, Spain
#+BIRTHDAY: 1990-05-15
#+LANGUAGE: en es ca
#+PINNED: 2025-01-15T10:00:00+0100
* Posts
** 2025-01-15T10:00:00+0100
:PROPERTIES:
:END:
This is my pinned post.
** 2025-01-16T12:00:00+0100
:PROPERTIES:
:END:
This is a regular post.
"""
# When: We parse the content
from app.feeds.parser import parse_org_social_content
result = parse_org_social_content(content)
# Then: All v1.6 metadata fields should be correctly parsed
self.assertEqual(result["metadata"]["location"], "Valencia, Spain")
self.assertEqual(result["metadata"]["birthday"], "1990-05-15")
self.assertEqual(result["metadata"]["language"], "en es ca")
self.assertEqual(result["metadata"]["pinned"], "2025-01-15T10:00:00+0100")
# Then: Posts should be parsed correctly
self.assertEqual(len(result["posts"]), 2)
def test_parse_invalid_birthday_is_dropped(self):
"""An invalid birthday format must be dropped, not abort the feed."""
# Given: A feed with a birthday that is not in YYYY-MM-DD format
content = """#+TITLE: Test User Profile
#+NICK: test_user
#+BIRTHDAY: 2003/06/17
* Posts
** 2025-01-15T10:00:00+0100
:PROPERTIES:
:END:
A post.
"""
# When: We parse the content
from app.feeds.parser import parse_org_social_content
result = parse_org_social_content(content)
# Then: The malformed birthday is dropped but the feed is still parsed
self.assertEqual(result["metadata"]["birthday"], "")
self.assertEqual(result["metadata"]["nick"], "test_user")
self.assertEqual(len(result["posts"]), 1)
def test_parse_post_id_in_header(self):
"""Test parsing post ID from header (v1.6 feature)."""
# Given: An org social content with post ID in header
content = """#+TITLE: Test
#+NICK: test_user
* Posts
** 2025-01-15T10:00:00+0100
:PROPERTIES:
:LANG: en
:END:
This post has ID in the header.
**
:PROPERTIES:
:ID: 2025-01-16T12:00:00+0100
:END:
This post has ID in properties.
"""
# When: We parse the content
from app.feeds.parser import parse_org_social_content
result = parse_org_social_content(content)
# Then: Both posts should have correct IDs
self.assertEqual(len(result["posts"]), 2)
# Then: First post should have ID from header
post1 = result["posts"][0]
self.assertEqual(post1["id"], "2025-01-15T10:00:00+0100")
# Then: Second post should have ID from properties
post2 = result["posts"][1]
self.assertEqual(post2["id"], "2025-01-16T12:00:00+0100")
def test_parse_post_id_priority_header_over_property(self):
"""Test that header ID takes priority over property ID (v1.6 spec)."""
# Given: An org social content with post ID in both header and properties
content = """#+TITLE: Test
#+NICK: test_user
* Posts
** 2025-01-15T10:00:00+0100
:PROPERTIES:
:ID: 2025-01-16T12:00:00+0100
:END:
This post has ID in both places. Header should take priority.
"""
# When: We parse the content
from app.feeds.parser import parse_org_social_content
result = parse_org_social_content(content)
# Then: Post should have ID from header (priority)
self.assertEqual(len(result["posts"]), 1)
post = result["posts"][0]
self.assertEqual(post["id"], "2025-01-15T10:00:00+0100")
@patch("app.feeds.parser.requests.get")
def test_parse_feed_with_301_redirect(self, mock_get):
"""Test that feed redirects (301) are properly detected and handled."""
# Given: A feed URL that redirects to a new URL
old_url = "https://old.example.com/social.org"
new_url = "https://new.example.com/social.org"
content = """#+TITLE: Test User
#+NICK: test_user
* Posts
** 2025-01-15T10:00:00+0100
:PROPERTIES:
:END:
Test post
"""
# Given: Mock response with redirect history
mock_redirect = Mock()
mock_redirect.status_code = 301
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = content.encode("utf-8")
mock_response.url = new_url # Final URL after redirect
mock_response.history = [mock_redirect] # Redirect happened
mock_response.raise_for_status = Mock()
mock_get.return_value = mock_response
# When: We parse the feed from old URL
from app.feeds.parser import parse_org_social
result = parse_org_social(old_url)
# Then: The content should be parsed correctly
self.assertEqual(result["metadata"]["nick"], "test_user")
self.assertEqual(len(result["posts"]), 1)
@patch("app.feeds.parser.requests.get")
def test_validate_feed_with_301_redirect(self, mock_get):
"""Test that feed validation handles redirects properly."""
# Given: A feed URL that redirects to a new URL
old_url = "https://old.example.com/social.org"
new_url = "https://new.example.com/social.org"
content = """#+TITLE: Test User
#+NICK: test_user
* Posts
"""
# Given: Mock response with redirect history
mock_redirect = Mock()
mock_redirect.status_code = 301
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = content.encode("utf-8")
mock_response.url = new_url # Final URL after redirect
mock_response.history = [mock_redirect] # Redirect happened
mock_response.raise_for_status = Mock()
mock_get.return_value = mock_response
# When: We validate the feed
from app.feeds.parser import validate_org_social_feed
is_valid, error_message = validate_org_social_feed(old_url)
# Then: The feed should be valid
self.assertTrue(is_valid)
self.assertEqual(error_message, "")
class FeedRedirectHandlerTest(TestCase):
"""Test cases for feed redirect handling logic."""
def test_handle_redirect_updates_feed_url_when_only_old_exists(self):
"""Test updating feed URL when only old feed exists."""
# Given: A feed exists with old URL
from app.feeds.models import Feed, Profile
from app.feeds.parser import _handle_feed_redirect
old_url = "https://old.example.com/social.org"
new_url = "https://new.example.com/social.org"
Feed.objects.create(url=old_url)
Profile.objects.create(
feed=old_url, nick="testuser", title="Test User", version="v1"
)
# When: Redirect is handled
_handle_feed_redirect(old_url, new_url)
# Then: Feed URL should be updated
self.assertFalse(Feed.objects.filter(url=old_url).exists())
self.assertTrue(Feed.objects.filter(url=new_url).exists())
# Then: Profile should point to new URL
profile = Profile.objects.get(feed=new_url)
self.assertEqual(profile.nick, "testuser")
def test_handle_redirect_merges_feeds_when_both_exist(self):
"""Test merging feeds when both old and new URLs exist."""
# Given: Both old and new feeds exist
from app.feeds.models import Feed, Profile, Post
from app.feeds.parser import _handle_feed_redirect
from django.utils import timezone
old_url = "https://old.example.com/social.org"
new_url = "https://new.example.com/social.org"
# Create old feed and profile
Feed.objects.create(url=old_url)
old_profile = Profile.objects.create(
feed=old_url, nick="olduser", title="Old User", version="v1"
)
Post.objects.create(
profile=old_profile,
post_id="2025-01-15T10:00:00+0100",
content="Old post",
created_at=timezone.now(),
)
# Create new feed and profile
Feed.objects.create(url=new_url)
new_profile = Profile.objects.create(
feed=new_url, nick="newuser", title="New User", version="v2"
)
# When: Redirect is handled
_handle_feed_redirect(old_url, new_url)
# Then: Old feed should be deleted
self.assertFalse(Feed.objects.filter(url=old_url).exists())
self.assertTrue(Feed.objects.filter(url=new_url).exists())
# Then: Old profile should be deleted
self.assertFalse(Profile.objects.filter(feed=old_url).exists())
# Then: Post should be migrated to new profile
post = Post.objects.get(post_id="2025-01-15T10:00:00+0100")
self.assertEqual(post.profile, new_profile)
def test_handle_redirect_merges_mentions_without_duplicates(self):
"""Test that mentions are merged correctly, avoiding UNIQUE constraint errors."""
# Given: Both old and new profiles exist with mentions
from app.feeds.models import Feed, Profile, Post, Mention
from app.feeds.parser import _handle_feed_redirect
from django.utils import timezone
old_url = "https://old.example.com/social.org"
new_url = "https://new.example.com/social.org"
# Create old feed and profile
Feed.objects.create(url=old_url)
old_profile = Profile.objects.create(
feed=old_url, nick="olduser", title="Old User", version="v1"
)
# Create new feed and profile
Feed.objects.create(url=new_url)
new_profile = Profile.objects.create(
feed=new_url, nick="newuser", title="New User", version="v2"
)
# Create a third profile that mentions both old and new profiles
third_profile = Profile.objects.create(
feed="https://third.example.com/social.org",
nick="third",
title="Third User",
version="v1",
)
post = Post.objects.create(
profile=third_profile,
post_id="2025-01-15T10:00:00+0100",
content="Mentioning both profiles",
created_at=timezone.now(),
)
# Create mentions: one to old_profile, one to new_profile
Mention.objects.create(post=post, mentioned_profile=old_profile, nickname="old")
Mention.objects.create(post=post, mentioned_profile=new_profile, nickname="new")
# When: Redirect is handled (this should NOT raise UNIQUE constraint error)
_handle_feed_redirect(old_url, new_url)
# Then: Old profile should be deleted
self.assertFalse(Profile.objects.filter(feed=old_url).exists())
# Then: Only one mention should remain (the duplicate was deleted)
mentions = Mention.objects.filter(post=post)
self.assertEqual(mentions.count(), 1)
self.assertEqual(mentions.first().mentioned_profile, new_profile)

217
app/feeds/test_scan_feeds.py Normal file
View file

@ -0,0 +1,217 @@
from unittest.mock import Mock, patch
from django.test import TestCase
from app.feeds.models import Feed, Profile, Post
from app.feeds.tasks import scan_feeds
class PostDeletionDetectionTest(TestCase):
"""Test cases for detecting and removing deleted posts."""
def setUp(self):
"""Set up test fixtures."""
self.feed_url = "https://example.com/social.org"
self.feed = Feed.objects.create(url=self.feed_url)
self.profile = Profile.objects.create(
feed=self.feed_url,
nick="test_user",
title="Test User",
)
def test_detect_deleted_posts(self):
"""Test deletion detection logic."""
# Given: Profile has 3 posts in database
Post.objects.create(
profile=self.profile, post_id="2025-01-01T10:00:00+0100", content="Post 1"
)
Post.objects.create(
profile=self.profile, post_id="2025-01-02T10:00:00+0100", content="Post 2"
)
Post.objects.create(
profile=self.profile, post_id="2025-01-03T10:00:00+0100", content="Post 3"
)
# When: Current feed only has 2 posts (simulating one was deleted)
current_post_ids = {"2025-01-01T10:00:00+0100", "2025-01-03T10:00:00+0100"}
# Get existing posts from database
existing_posts = Post.objects.filter(profile=self.profile)
existing_post_ids = set(existing_posts.values_list("post_id", flat=True))
# Find deleted posts
deleted_post_ids = existing_post_ids - current_post_ids
# Then: Should detect one deleted post
self.assertEqual(len(deleted_post_ids), 1)
self.assertIn("2025-01-02T10:00:00+0100", deleted_post_ids)
# When: Delete the posts that no longer exist
deleted_count = Post.objects.filter(
profile=self.profile, post_id__in=deleted_post_ids
).delete()[0]
# Then: One post should be deleted
self.assertEqual(deleted_count, 1)
# Then: Only 2 posts should remain
remaining_posts = Post.objects.filter(profile=self.profile)
self.assertEqual(remaining_posts.count(), 2)
# Then: Deleted post should not exist
self.assertFalse(
Post.objects.filter(
profile=self.profile, post_id="2025-01-02T10:00:00+0100"
).exists()
)
# Then: Other posts should still exist
self.assertTrue(
Post.objects.filter(
profile=self.profile, post_id="2025-01-01T10:00:00+0100"
).exists()
)
self.assertTrue(
Post.objects.filter(
profile=self.profile, post_id="2025-01-03T10:00:00+0100"
).exists()
)
def test_no_posts_deleted_when_all_present(self):
"""Test that no posts are deleted when all posts are still in feed."""
# Given: Profile has 2 posts
Post.objects.create(
profile=self.profile, post_id="2025-01-01T10:00:00+0100", content="Post 1"
)
Post.objects.create(
profile=self.profile, post_id="2025-01-02T10:00:00+0100", content="Post 2"
)
# When: Current feed still has both posts
current_post_ids = {"2025-01-01T10:00:00+0100", "2025-01-02T10:00:00+0100"}
# Get existing posts
existing_posts = Post.objects.filter(profile=self.profile)
existing_post_ids = set(existing_posts.values_list("post_id", flat=True))
# Find deleted posts
deleted_post_ids = existing_post_ids - current_post_ids
# Then: No posts should be detected as deleted
self.assertEqual(len(deleted_post_ids), 0)
# Then: Both posts should still exist
self.assertEqual(Post.objects.filter(profile=self.profile).count(), 2)
def test_all_posts_deleted(self):
"""Test handling when all posts are deleted from feed."""
# Given: Profile has 2 posts
Post.objects.create(
profile=self.profile, post_id="2025-01-01T10:00:00+0100", content="Post 1"
)
Post.objects.create(
profile=self.profile, post_id="2025-01-02T10:00:00+0100", content="Post 2"
)
# When: Current feed has no posts
current_post_ids = set()
# Get existing posts
existing_posts = Post.objects.filter(profile=self.profile)
existing_post_ids = set(existing_posts.values_list("post_id", flat=True))
# Find deleted posts
deleted_post_ids = existing_post_ids - current_post_ids
# Then: All posts should be detected as deleted
self.assertEqual(len(deleted_post_ids), 2)
# When: Delete all posts
deleted_count = Post.objects.filter(
profile=self.profile, post_id__in=deleted_post_ids
).delete()[0]
# Then: 2 posts should be deleted
self.assertEqual(deleted_count, 2)
# Then: No posts should remain
self.assertEqual(Post.objects.filter(profile=self.profile).count(), 0)
def test_multiple_posts_deleted(self):
"""Test handling when multiple posts are deleted."""
# Given: Profile has 5 posts
for i in range(1, 6):
Post.objects.create(
profile=self.profile,
post_id=f"2025-01-0{i}T10:00:00+0100",
content=f"Post {i}",
)
# When: Current feed only has 2 posts (3 were deleted)
current_post_ids = {"2025-01-01T10:00:00+0100", "2025-01-05T10:00:00+0100"}
# Get existing posts
existing_posts = Post.objects.filter(profile=self.profile)
existing_post_ids = set(existing_posts.values_list("post_id", flat=True))
# Find deleted posts
deleted_post_ids = existing_post_ids - current_post_ids
# Then: 3 posts should be detected as deleted
self.assertEqual(len(deleted_post_ids), 3)
# When: Delete them
deleted_count = Post.objects.filter(
profile=self.profile, post_id__in=deleted_post_ids
).delete()[0]
# Then: 3 posts should be deleted
self.assertEqual(deleted_count, 3)
# Then: Only 2 posts should remain
self.assertEqual(Post.objects.filter(profile=self.profile).count(), 2)
class ScanFeedsRobustnessTest(TestCase):
"""End-to-end robustness tests for the scan_feeds task."""
@patch("app.feeds.parser.requests.get")
def test_invalid_birthday_does_not_abort_scan(self, mock_get):
"""A feed with a malformed birthday must still be scanned (regression).
Previously a value like "2003/06/17" reached the Profile.birthday
DateField and raised a ValidationError, aborting the whole feed scan on
every run. The birthday must now be dropped while posts are still saved.
"""
# Given: A feed whose birthday is not in YYYY-MM-DD format
feed_url = "https://host.example.org/ali/social.org"
Feed.objects.create(url=feed_url)
content = (
"#+TITLE: Ali\n"
"#+NICK: ali\n"
"#+BIRTHDAY: 2003/06/17\n"
"\n"
"* Posts\n"
"** 2025-01-01T10:00:00+0100\n"
":PROPERTIES:\n"
":END:\n"
"\n"
"Hello world\n"
)
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = content.encode("utf-8")
mock_response.url = feed_url # No redirect
mock_response.history = []
mock_response.raise_for_status = Mock()
mock_get.return_value = mock_response
# When: We scan all feeds
scan_feeds.call_local()
# Then: The profile is created with the bad birthday dropped, posts saved
profile = Profile.objects.get(feed=feed_url)
self.assertEqual(profile.nick, "ali")
self.assertIsNone(profile.birthday)
self.assertEqual(Post.objects.filter(profile=profile).count(), 1)

656
app/feeds/test_webmentions.py Normal file
View file

@ -0,0 +1,656 @@
from datetime import timedelta
from unittest.mock import Mock, patch
import requests
from django.test import TestCase
from django.utils import timezone
from app.feeds.models import Feed, OutgoingWebmention, Profile
from app.feeds.tasks import WEBMENTION_MAX_ATTEMPTS, _send_pending_webmentions_impl
from app.feeds.webmentions import (
discover_webmention_endpoint,
extract_external_urls,
is_safe_endpoint,
queue_webmentions_for_post,
)
class ExtractExternalUrlsTest(TestCase):
"""Test cases for URL extraction from post content."""
def test_extract_org_link_with_description(self):
# Given: A post containing an Org link with description
content = "Great read: [[https://example.com/article][My article]]"
# When: URLs are extracted
result = extract_external_urls(content)
# Then: The linked URL is found without the description
self.assertEqual(result, ["https://example.com/article"])
def test_extract_org_link_without_description(self):
# Given: A post containing a plain Org link
content = "See [[https://example.com/article]] for details"
# When: URLs are extracted
result = extract_external_urls(content)
# Then: The linked URL is found
self.assertEqual(result, ["https://example.com/article"])
def test_extract_bare_url(self):
# Given: A post containing a bare URL
content = "Check https://example.com/post and tell me"
# When: URLs are extracted
result = extract_external_urls(content)
# Then: The bare URL is found
self.assertEqual(result, ["https://example.com/post"])
def test_bare_url_trailing_punctuation_stripped(self):
# Given: A bare URL wrapped in punctuation
content = "Nice one (https://example.com/post). Really!"
# When: URLs are extracted
result = extract_external_urls(content)
# Then: Trailing punctuation is not part of the URL
self.assertEqual(result, ["https://example.com/post"])
def test_url_with_query_string_and_fragment_is_preserved(self):
# Given: A URL with query string and fragment
content = "See https://example.com/post?id=1&lang=en#section-2"
# When: URLs are extracted
result = extract_external_urls(content)
# Then: Query string and fragment are kept intact
self.assertEqual(result, ["https://example.com/post?id=1&lang=en#section-2"])
def test_org_social_mentions_are_ignored(self):
# Given: A post with an Org Social mention and a normal link
content = (
"Hi [[org-social:https://other.example/social.org][bob]] look at "
"[[https://example.com/article][this]]"
)
# When: URLs are extracted
result = extract_external_urls(content)
# Then: Only the normal link is found, the mention is skipped
self.assertEqual(result, ["https://example.com/article"])
def test_duplicate_urls_are_deduplicated(self):
# Given: A post repeating the same URL as Org link and bare URL
content = (
"https://example.com/a and again [[https://example.com/a][same]] "
"plus https://example.com/b"
)
# When: URLs are extracted
result = extract_external_urls(content)
# Then: Each URL appears once, in order of appearance
self.assertEqual(result, ["https://example.com/a", "https://example.com/b"])
def test_empty_or_missing_content_returns_no_urls(self):
# Given: Empty and missing content
# When: URLs are extracted
# Then: No URLs are found and nothing breaks
self.assertEqual(extract_external_urls(""), [])
self.assertEqual(extract_external_urls(None), [])
def test_non_http_schemes_are_ignored(self):
# Given: A post with non-http(s) URIs only
content = "Write to mailto:me@example.com or ftp://example.com/file"
# When: URLs are extracted
result = extract_external_urls(content)
# Then: Nothing is extracted
self.assertEqual(result, [])
def _mock_response(
url="https://example.com/article",
headers=None,
body=b"",
encoding="utf-8",
):
"""Build a minimal mock of a streamed requests.Response."""
response = Mock()
response.url = url
response.headers = headers or {}
response.encoding = encoding
response.raw = Mock()
response.raw.read = Mock(return_value=body)
return response
class DiscoverWebmentionEndpointTest(TestCase):
"""Test cases for endpoint discovery per the W3C spec."""
@patch("app.feeds.webmentions.requests.get")
def test_link_header_discovery(self, mock_get):
# Given: A target advertising its endpoint in the HTTP Link header
mock_get.return_value = _mock_response(
headers={
"Link": '<https://example.com/wm>; rel="webmention"',
"Content-Type": "text/html",
}
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: The Link header endpoint is returned
self.assertEqual(endpoint, "https://example.com/wm")
@patch("app.feeds.webmentions.requests.get")
def test_link_header_takes_precedence_over_html(self, mock_get):
# Given: A target with endpoints both in the Link header and the HTML
mock_get.return_value = _mock_response(
headers={
"Link": '<https://example.com/header-wm>; rel="webmention"',
"Content-Type": "text/html",
},
body=b'<link rel="webmention" href="https://example.com/html-wm">',
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: The Link header wins, as the spec requires
self.assertEqual(endpoint, "https://example.com/header-wm")
@patch("app.feeds.webmentions.requests.get")
def test_link_header_with_multiple_rels(self, mock_get):
# Given: A Link header whose rel lists several values
mock_get.return_value = _mock_response(
headers={
"Link": '<https://example.com/wm>; rel="webmention somethingelse"',
"Content-Type": "text/html",
}
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: The webmention rel is still recognized
self.assertEqual(endpoint, "https://example.com/wm")
@patch("app.feeds.webmentions.requests.get")
def test_html_link_element_discovery(self, mock_get):
# Given: A target advertising a relative endpoint in a <link> element
mock_get.return_value = _mock_response(
headers={"Content-Type": "text/html; charset=utf-8"},
body=b'<html><head><link rel="webmention" href="/wm"></head></html>',
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: The relative href is resolved against the target
self.assertEqual(endpoint, "https://example.com/wm")
@patch("app.feeds.webmentions.requests.get")
def test_html_anchor_element_discovery(self, mock_get):
# Given: A target advertising its endpoint in an <a> element
mock_get.return_value = _mock_response(
headers={"Content-Type": "text/html"},
body=b'<body><a rel="webmention" href="https://wm.example.org/ep">wm</a></body>',
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: The anchor endpoint is returned
self.assertEqual(endpoint, "https://wm.example.org/ep")
@patch("app.feeds.webmentions.requests.get")
def test_first_element_in_document_order_wins(self, mock_get):
# Given: Both an <a> and a <link> endpoint, the <a> appearing first
mock_get.return_value = _mock_response(
headers={"Content-Type": "text/html"},
body=(
b'<a rel="webmention" href="/first">a</a>'
b'<link rel="webmention" href="/second">'
),
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: The first element in document order wins
self.assertEqual(endpoint, "https://example.com/first")
@patch("app.feeds.webmentions.requests.get")
def test_empty_href_resolves_to_page_url(self, mock_get):
# Given: An endpoint advertised with an empty href
mock_get.return_value = _mock_response(
url="https://example.com/article",
headers={"Content-Type": "text/html"},
body=b'<link rel="webmention" href="">',
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: It resolves to the page URL itself
self.assertEqual(endpoint, "https://example.com/article")
@patch("app.feeds.webmentions.requests.get")
def test_relative_endpoint_resolved_after_redirect(self, mock_get):
# Given: A target that redirected and advertises a relative endpoint
mock_get.return_value = _mock_response(
url="https://final.example.com/page",
headers={"Content-Type": "text/html"},
body=b'<link rel="webmention" href="wm-endpoint">',
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: The endpoint resolves against the final URL after redirects
self.assertEqual(endpoint, "https://final.example.com/wm-endpoint")
@patch("app.feeds.webmentions.requests.get")
def test_no_endpoint_returns_none(self, mock_get):
# Given: A target without any webmention endpoint
mock_get.return_value = _mock_response(
headers={"Content-Type": "text/html"},
body=b"<html><body>No webmention here</body></html>",
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: No endpoint is found
self.assertIsNone(endpoint)
@patch("app.feeds.webmentions.requests.get")
def test_non_html_content_without_link_header_returns_none(self, mock_get):
# Given: A non-HTML target without a Link header
mock_get.return_value = _mock_response(
headers={"Content-Type": "application/pdf"},
body=b"%PDF-1.4",
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/doc.pdf")
# Then: No endpoint is found and the body is never parsed as HTML
self.assertIsNone(endpoint)
@patch("app.feeds.webmentions.requests.get")
def test_rel_without_webmention_is_ignored(self, mock_get):
# Given: A page whose only <link> has an unrelated rel
mock_get.return_value = _mock_response(
headers={"Content-Type": "text/html"},
body=b'<link rel="stylesheet" href="/style.css">',
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: No endpoint is found
self.assertIsNone(endpoint)
class IsSafeEndpointTest(TestCase):
"""Test cases for endpoint safety checks (spec section 4.3)."""
def test_rejects_non_http_scheme(self):
# Given: Endpoints with non-http(s) schemes
# When: They are checked
# Then: They are rejected
self.assertFalse(is_safe_endpoint("ftp://example.com/wm"))
self.assertFalse(is_safe_endpoint("file:///etc/passwd"))
@patch("app.feeds.webmentions.socket.getaddrinfo")
def test_rejects_loopback(self, mock_getaddrinfo):
# Given: An endpoint resolving to a loopback address
mock_getaddrinfo.return_value = [(2, 1, 6, "", ("127.0.0.1", 80))]
# When/Then: It is rejected
self.assertFalse(is_safe_endpoint("http://localhost/wm"))
@patch("app.feeds.webmentions.socket.getaddrinfo")
def test_rejects_private_address(self, mock_getaddrinfo):
# Given: An endpoint resolving to a private address
mock_getaddrinfo.return_value = [(2, 1, 6, "", ("192.168.1.10", 80))]
# When/Then: It is rejected
self.assertFalse(is_safe_endpoint("https://internal.example.com/wm"))
@patch("app.feeds.webmentions.socket.getaddrinfo")
def test_accepts_public_address(self, mock_getaddrinfo):
# Given: An endpoint resolving to a public address
mock_getaddrinfo.return_value = [(2, 1, 6, "", ("93.184.216.34", 443))]
# When/Then: It is accepted
self.assertTrue(is_safe_endpoint("https://example.com/wm"))
@patch("app.feeds.webmentions.socket.getaddrinfo")
def test_rejects_unresolvable_host(self, mock_getaddrinfo):
# Given: An endpoint whose host does not resolve
import socket as socket_module
mock_getaddrinfo.side_effect = socket_module.gaierror()
# When/Then: It is rejected
self.assertFalse(is_safe_endpoint("https://nope.invalid/wm"))
class QueueWebmentionsForPostTest(TestCase):
"""Test cases for queueing outgoing webmentions during feed scans."""
def setUp(self):
"""Set up test fixtures."""
self.feed_url = "https://example.com/social.org"
self.feed = Feed.objects.create(url=self.feed_url)
self.profile = Profile.objects.create(
feed=self.feed_url, nick="alice", title="Alice"
)
self.post_id = "2026-01-01T10:00:00+00:00"
def test_queues_external_urls_as_pending(self):
# Given: A post linking to an external article
content = "I wrote about it: [[https://blog.example.org/post][my post]]"
# When: Webmentions are queued for the post
queued = queue_webmentions_for_post(self.profile, self.post_id, content)
# Then: One pending webmention exists with the post as source
self.assertEqual(queued, 1)
webmention = OutgoingWebmention.objects.get()
self.assertEqual(webmention.source, f"{self.feed_url}#{self.post_id}")
self.assertEqual(webmention.target, "https://blog.example.org/post")
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_PENDING)
def test_rescanning_same_post_does_not_duplicate(self):
# Given: A post whose webmention was already queued
content = "Look at https://blog.example.org/post please"
queue_webmentions_for_post(self.profile, self.post_id, content)
# When: The same post is queued again (rescan)
queued = queue_webmentions_for_post(self.profile, self.post_id, content)
# Then: Nothing new is queued
self.assertEqual(queued, 0)
self.assertEqual(OutgoingWebmention.objects.count(), 1)
def test_rescan_does_not_reset_sent_status(self):
# Given: A webmention already delivered for the post
content = "Look at https://blog.example.org/post please"
queue_webmentions_for_post(self.profile, self.post_id, content)
OutgoingWebmention.objects.update(status=OutgoingWebmention.STATUS_SENT)
# When: The same post is queued again (rescan)
queue_webmentions_for_post(self.profile, self.post_id, content)
# Then: The delivered webmention keeps its sent status
self.assertEqual(
OutgoingWebmention.objects.get().status, OutgoingWebmention.STATUS_SENT
)
def test_edited_post_queues_only_new_urls(self):
# Given: A post whose original link was already delivered
queue_webmentions_for_post(
self.profile, self.post_id, "First link https://blog.example.org/a"
)
OutgoingWebmention.objects.update(status=OutgoingWebmention.STATUS_SENT)
# When: The post is edited adding a second link
queued = queue_webmentions_for_post(
self.profile,
self.post_id,
"First link https://blog.example.org/a and https://blog.example.org/b",
)
# Then: Only the added link is queued, the delivered one is untouched
self.assertEqual(queued, 1)
self.assertEqual(OutgoingWebmention.objects.count(), 2)
self.assertEqual(
OutgoingWebmention.objects.get(target="https://blog.example.org/a").status,
OutgoingWebmention.STATUS_SENT,
)
def test_registered_feeds_are_excluded(self):
# Given: A post linking to a registered Org Social feed and to an article
Feed.objects.create(url="https://friend.example.org/social.org")
content = (
"Read https://friend.example.org/social.org#2026-01-01T00:00:00+00:00 "
"and https://blog.example.org/post"
)
# When: Webmentions are queued for the post
queued = queue_webmentions_for_post(self.profile, self.post_id, content)
# Then: Only the article is queued, the feed link is handled natively
self.assertEqual(queued, 1)
self.assertEqual(
OutgoingWebmention.objects.get().target, "https://blog.example.org/post"
)
def test_own_feed_is_excluded(self):
# Given: A post linking to the author's own feed
content = f"Self reference {self.feed_url}#2025-12-31T00:00:00+00:00"
# When: Webmentions are queued for the post
queued = queue_webmentions_for_post(self.profile, self.post_id, content)
# Then: Nothing is queued
self.assertEqual(queued, 0)
def test_post_without_urls_queues_nothing(self):
# Given: A post with no URLs at all
content = "Just text"
# When: Webmentions are queued for the post
queued = queue_webmentions_for_post(self.profile, self.post_id, content)
# Then: Nothing is queued
self.assertEqual(queued, 0)
self.assertEqual(OutgoingWebmention.objects.count(), 0)
class ScanFeedsWebmentionIntegrationTest(TestCase):
"""Integration tests: scan_feeds queues webmentions exactly once."""
def setUp(self):
"""Set up test fixtures."""
self.feed_url = "https://example.com/social.org"
self.feed = Feed.objects.create(url=self.feed_url)
def _parsed_feed(self, content):
return {
"metadata": {"title": "Alice", "nick": "alice"},
"posts": [
{
"id": "2026-01-01T10:00:00+00:00",
"content": content,
"properties": {},
"mentions": [],
"poll_options": [],
}
],
}
@patch("app.feeds.parser.parse_org_social")
def test_new_post_queues_webmention_once(self, mock_parse):
from app.feeds.tasks import scan_feeds
# Given: A feed with a new post linking to an article
mock_parse.return_value = self._parsed_feed(
"Article: https://blog.example.org/post"
)
# When: The feed is scanned twice without changes
scan_feeds.call_local()
first_scan_count = OutgoingWebmention.objects.count()
scan_feeds.call_local()
# Then: Exactly one webmention is queued, the rescan adds nothing
self.assertEqual(first_scan_count, 1)
self.assertEqual(OutgoingWebmention.objects.count(), 1)
@patch("app.feeds.parser.parse_org_social")
def test_edited_post_queues_only_added_link(self, mock_parse):
from app.feeds.tasks import scan_feeds
# Given: A scanned feed whose post links to one article
mock_parse.return_value = self._parsed_feed(
"Article: https://blog.example.org/a"
)
scan_feeds.call_local()
self.assertEqual(OutgoingWebmention.objects.count(), 1)
# When: The post is edited adding a second link and rescanned
mock_parse.return_value = self._parsed_feed(
"Article: https://blog.example.org/a and https://blog.example.org/b"
)
scan_feeds.call_local()
# Then: Only the added link produced a new webmention
self.assertEqual(OutgoingWebmention.objects.count(), 2)
targets = set(OutgoingWebmention.objects.values_list("target", flat=True))
self.assertEqual(
targets, {"https://blog.example.org/a", "https://blog.example.org/b"}
)
class SendPendingWebmentionsTest(TestCase):
"""Test cases for the delivery task."""
def _create_webmention(self, **kwargs):
defaults = {
"source": "https://example.com/social.org#2026-01-01T10:00:00+00:00",
"target": "https://blog.example.org/post",
}
defaults.update(kwargs)
return OutgoingWebmention.objects.create(**defaults)
@patch("app.feeds.webmentions.is_safe_endpoint", return_value=True)
@patch("app.feeds.webmentions.send_webmention")
@patch("app.feeds.webmentions.discover_webmention_endpoint")
def test_pending_webmention_is_sent(self, mock_discover, mock_send, _mock_safe):
# Given: A pending webmention whose target accepts it with 202
mock_discover.return_value = "https://blog.example.org/wm"
mock_send.return_value = 202
webmention = self._create_webmention()
# When: The delivery task runs
counters = _send_pending_webmentions_impl()
# Then: The webmention is delivered once and marked as sent
self.assertEqual(counters["sent"], 1)
webmention.refresh_from_db()
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_SENT)
self.assertEqual(webmention.endpoint, "https://blog.example.org/wm")
self.assertEqual(webmention.response_code, 202)
self.assertEqual(webmention.attempts, 1)
mock_send.assert_called_once_with(
"https://blog.example.org/wm", webmention.source, webmention.target
)
@patch("app.feeds.webmentions.discover_webmention_endpoint")
def test_target_without_endpoint_is_marked_permanently(self, mock_discover):
# Given: A pending webmention whose target has no endpoint
mock_discover.return_value = None
webmention = self._create_webmention()
# When: The delivery task runs
counters = _send_pending_webmentions_impl()
# Then: The webmention is marked permanently as no_endpoint
self.assertEqual(counters["no_endpoint"], 1)
webmention.refresh_from_db()
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_NO_ENDPOINT)
# Then: A later run never fetches that target again
mock_discover.reset_mock()
_send_pending_webmentions_impl()
mock_discover.assert_not_called()
@patch("app.feeds.webmentions.is_safe_endpoint", return_value=False)
@patch("app.feeds.webmentions.discover_webmention_endpoint")
def test_unsafe_endpoint_is_rejected(self, mock_discover, _mock_safe):
# Given: A pending webmention whose endpoint points to loopback
mock_discover.return_value = "http://127.0.0.1/wm"
webmention = self._create_webmention()
# When: The delivery task runs
counters = _send_pending_webmentions_impl()
# Then: Nothing is sent and the webmention is discarded
self.assertEqual(counters["no_endpoint"], 1)
webmention.refresh_from_db()
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_NO_ENDPOINT)
@patch("app.feeds.webmentions.discover_webmention_endpoint")
def test_network_error_marks_failed_and_retries_later(self, mock_discover):
# Given: A pending webmention whose target is unreachable
mock_discover.side_effect = requests.RequestException("boom")
webmention = self._create_webmention()
# When: The delivery task runs
counters = _send_pending_webmentions_impl()
# Then: The webmention is marked failed with one attempt
self.assertEqual(counters["failed"], 1)
webmention.refresh_from_db()
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_FAILED)
self.assertEqual(webmention.attempts, 1)
# Then: Within the backoff window it is skipped
mock_discover.reset_mock()
_send_pending_webmentions_impl()
mock_discover.assert_not_called()
# Then: After the backoff window it is retried
webmention.last_attempt_at = timezone.now() - timedelta(hours=2)
webmention.save()
mock_discover.side_effect = None
mock_discover.return_value = None
_send_pending_webmentions_impl()
webmention.refresh_from_db()
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_NO_ENDPOINT)
@patch("app.feeds.webmentions.discover_webmention_endpoint")
def test_max_attempts_reached_stops_retrying(self, mock_discover):
# Given: A webmention that already exhausted its attempts long ago
self._create_webmention(
status=OutgoingWebmention.STATUS_FAILED,
attempts=WEBMENTION_MAX_ATTEMPTS,
last_attempt_at=timezone.now() - timedelta(days=30),
)
# When: The delivery task runs
counters = _send_pending_webmentions_impl()
# Then: It is never processed again
self.assertEqual(counters, {"sent": 0, "no_endpoint": 0, "failed": 0})
mock_discover.assert_not_called()
@patch("app.feeds.webmentions.is_safe_endpoint", return_value=True)
@patch("app.feeds.webmentions.send_webmention")
@patch("app.feeds.webmentions.discover_webmention_endpoint")
def test_http_error_response_marks_failed(
self, mock_discover, mock_send, _mock_safe
):
# Given: A pending webmention whose endpoint answers HTTP 500
mock_discover.return_value = "https://blog.example.org/wm"
mock_send.return_value = 500
webmention = self._create_webmention()
# When: The delivery task runs
counters = _send_pending_webmentions_impl()
# Then: The webmention is marked failed keeping the endpoint for retry
self.assertEqual(counters["failed"], 1)
webmention.refresh_from_db()
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_FAILED)
self.assertEqual(webmention.response_code, 500)
self.assertEqual(webmention.endpoint, "https://blog.example.org/wm")

89
app/feeds/utils.py Normal file
View file

@ -0,0 +1,89 @@
"""
Utility functions for working with posts and feeds
"""
from typing import List
from app.feeds.models import Post, Profile
def get_parent_chain(post: Post, max_depth: int = 100) -> List[str]:
"""
Calculate the parent chain for a post.
Returns a list of post URLs from the root ancestor to the immediate parent.
For example, if the chain is: Alice → Bob → Carol → Dave → Current post
This returns: [Alice_URL, Bob_URL, Carol_URL, Dave_URL]
Args:
post: The Post object to calculate the chain for
max_depth: Maximum depth to traverse (prevents infinite loops)
Returns:
List of post URLs in order from root to immediate parent
"""
if not post.reply_to:
# This post is a root post (no parent)
return []
chain = []
current_reply_to = post.reply_to
depth = 0
while current_reply_to and depth < max_depth:
# Add current parent to chain
chain.append(current_reply_to)
# Parse the reply_to URL to find the parent post
try:
if "#" not in current_reply_to:
break
feed_url, post_id = current_reply_to.split("#", 1)
# Find the parent post
profile = Profile.objects.filter(feed=feed_url).first()
if not profile:
break
parent_post = Post.objects.filter(profile=profile, post_id=post_id).first()
if not parent_post:
break
# Move up to the next parent
current_reply_to = parent_post.reply_to
depth += 1
except (ValueError, AttributeError):
break
# Reverse the chain so it goes from root to immediate parent
chain.reverse()
return chain
def add_relay_headers(response):
"""
Add global ETag and Last-Modified headers to a response.
This ensures all endpoints return consistent caching headers.
The headers are retrieved from the RelayMetadata model, which is updated
by the scan_feeds task after each scan. This way:
- All endpoints return the same ETag and Last-Modified
- Headers are added even when serving cached responses
- The ETag changes only when feeds are actually scanned
Args:
response: Django REST framework Response object
Returns:
The same response object with headers added
"""
from app.feeds.models import RelayMetadata
etag, last_modified = RelayMetadata.get_global_metadata()
response["ETag"] = f'"{etag}"'
response["Last-Modified"] = last_modified.strftime("%a, %d %b %Y %H:%M:%S GMT")
return response

21
app/feeds/views.py
View file

@ -4,6 +4,8 @@ from rest_framework import status
from django.core.cache import cache
import logging
from app.bridge.models import bridge_urls_q, is_bridge_feed_url
from .models import Feed
from .parser import validate_org_social_feed
@ -32,8 +34,11 @@ class FeedsView(APIView):
status=status.HTTP_200_OK,
)
# If not in cache, query database
feeds = list(Feed.objects.all().values_list("url", flat=True))
# If not in cache, query database. Bridge virtual feeds are a
# connection helper, not real accounts, so they are never listed.
feeds = list(
Feed.objects.exclude(bridge_urls_q("url")).values_list("url", flat=True)
)
# Cache permanently (will be cleared by scan_feeds task)
cache.set(cache_key, feeds, None)
@ -62,6 +67,18 @@ class FeedsView(APIView):
feed_url = feed_url.strip()
# Bridge virtual feeds serve valid Org Social content but are not
# real accounts; they cannot be registered
if is_bridge_feed_url(feed_url):
return Response(
{
"type": "Error",
"errors": ["Bridge virtual feeds cannot be registered"],
"data": None,
},
status=status.HTTP_400_BAD_REQUEST,
)
# Check if feed already exists
existing_feed = Feed.objects.filter(url=feed_url).first()
if existing_feed:

202
app/feeds/webmentions.py Normal file
View file

@ -0,0 +1,202 @@
"""
Outgoing Webmention support (sender side only).
Implements the sender half of https://www.w3.org/TR/webmention/:
URL extraction from post content, endpoint discovery and notification.
Receiving webmentions is out of scope for the relay.
"""
import ipaddress
import logging
import re
import socket
from html.parser import HTMLParser
from urllib.parse import urljoin, urlparse
import requests
logger = logging.getLogger(__name__)
WEBMENTION_TIMEOUT = 10
WEBMENTION_USER_AGENT = "Org-Social-Relay (+https://relay.org-social.org/)"
# Maximum HTML bytes read while looking for a <link>/<a> endpoint
DISCOVERY_MAX_BYTES = 1024 * 1024
# Org Social mentions ([[org-social:url][nick]]) are handled natively by the
# relay and must never produce webmentions
ORG_SOCIAL_MENTION_RE = re.compile(r"\[\[org-social:[^\]]*\](?:\[[^\]]*\])?\]")
ORG_LINK_RE = re.compile(r"\[\[(https?://[^\]\[]+)\](?:\[[^\]]*\])?\]")
BARE_URL_RE = re.compile(r"https?://[^\s\]\[<>\"']+")
TRAILING_PUNCTUATION = ".,;:!?)'\""
def extract_external_urls(content):
"""
Extract http(s) URLs from post content, both Org links ([[url]] and
[[url][description]]) and bare URLs. Org Social mentions are ignored.
Returns a list of unique URLs in order of appearance.
"""
if not content:
return []
text = ORG_SOCIAL_MENTION_RE.sub(" ", content)
urls = []
for match in ORG_LINK_RE.finditer(text):
urls.append(match.group(1).strip())
# Remove Org link syntax so bare URL matching does not see them again
text = ORG_LINK_RE.sub(" ", text)
for match in BARE_URL_RE.finditer(text):
urls.append(match.group(0).rstrip(TRAILING_PUNCTUATION))
unique_urls = []
seen = set()
for url in urls:
if url and url not in seen:
seen.add(url)
unique_urls.append(url)
return unique_urls
class _EndpointHTMLParser(HTMLParser):
"""Finds the first <link> or <a> with rel="webmention" in document order."""
def __init__(self):
super().__init__(convert_charrefs=True)
self.endpoint = None
def handle_starttag(self, tag, attrs):
if self.endpoint is not None or tag not in ("link", "a"):
return
attrs_dict = dict(attrs)
rel = attrs_dict.get("rel") or ""
if "webmention" in rel.lower().split() and "href" in attrs_dict:
# An empty href is valid: it resolves to the page URL itself
self.endpoint = attrs_dict.get("href") or ""
def discover_webmention_endpoint(target_url):
"""
Discover the Webmention endpoint of a target URL following the spec
precedence: first HTTP Link header, then first <link>/<a> element in
document order. Relative endpoints are resolved against the final URL
after redirects.
Returns the absolute endpoint URL, or None if the target does not
advertise one. Raises requests.RequestException on network errors.
"""
response = requests.get(
target_url,
timeout=WEBMENTION_TIMEOUT,
headers={"User-Agent": WEBMENTION_USER_AGENT},
stream=True,
)
try:
link_header = response.headers.get("Link", "")
if link_header:
for link in requests.utils.parse_header_links(link_header):
rels = link.get("rel", "").lower().split()
if "webmention" in rels and "url" in link:
return urljoin(response.url, link["url"])
content_type = response.headers.get("Content-Type", "")
if "html" not in content_type.lower():
return None
raw_content = response.raw.read(DISCOVERY_MAX_BYTES, decode_content=True)
html = raw_content.decode(response.encoding or "utf-8", errors="replace")
finally:
response.close()
parser = _EndpointHTMLParser()
parser.feed(html)
if parser.endpoint is None:
return None
return urljoin(response.url, parser.endpoint)
def is_safe_endpoint(endpoint_url):
"""
Reject endpoints that are not plain http(s) or that resolve to loopback,
private, link-local or otherwise non-global addresses (spec section 4.3).
"""
parsed = urlparse(endpoint_url)
if parsed.scheme not in ("http", "https") or not parsed.hostname:
return False
try:
addr_info = socket.getaddrinfo(parsed.hostname, None)
except (socket.gaierror, UnicodeError):
return False
for info in addr_info:
try:
address = ipaddress.ip_address(info[4][0])
except ValueError:
return False
if (
address.is_loopback
or address.is_private
or address.is_link_local
or address.is_multicast
or address.is_reserved
or address.is_unspecified
):
return False
return True
def send_webmention(endpoint_url, source, target):
"""
POST the webmention as application/x-www-form-urlencoded and return the
HTTP status code. Raises requests.RequestException on network errors.
"""
response = requests.post(
endpoint_url,
data={"source": source, "target": target},
timeout=WEBMENTION_TIMEOUT,
headers={"User-Agent": WEBMENTION_USER_AGENT},
)
return response.status_code
def queue_webmentions_for_post(profile, post_id, content):
"""
Create pending OutgoingWebmention rows for every external URL in a post.
URLs pointing to registered Org Social feeds (or the author's own feed)
are skipped: the relay already handles those interactions natively.
Existing (source, target) pairs are never touched, so a webmention is
sent at most once even across rescans and post edits.
Returns the number of newly queued webmentions.
"""
from .models import Feed, OutgoingWebmention
urls = extract_external_urls(content)
if not urls:
return 0
source = f"{profile.feed}#{post_id}"
base_urls = {url.split("#")[0] for url in urls}
known_feeds = set(
Feed.objects.filter(url__in=base_urls).values_list("url", flat=True)
)
queued = 0
for url in urls:
base_url = url.split("#")[0]
if base_url == profile.feed or base_url in known_feeds:
continue
_, created = OutgoingWebmention.objects.get_or_create(source=source, target=url)
if created:
queued += 1
if queued:
logger.info(f"Queued {queued} webmention(s) for {source}")
return queued

11
app/groups/tests.py
View file

@ -119,7 +119,10 @@ class GroupMessagesViewTest(TestCase):
# Check meta - should contain display name, not slug
self.assertEqual(response.data["meta"]["group"], "Emacs")
self.assertIn("members", response.data["meta"])
self.assertIn("version", response.data["meta"])
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
@override_settings(ENABLED_GROUPS=["emacs"], GROUPS_MAP={"emacs": "Emacs"})
def test_get_group_messages_with_replies(self):
@ -206,10 +209,8 @@ class GroupMessagesViewTest(TestCase):
self.assertEqual(response1.status_code, status.HTTP_200_OK)
self.assertEqual(response2.status_code, status.HTTP_200_OK)
# And version should be the same (indicating cache hit)
self.assertEqual(
response1.data["meta"]["version"], response2.data["meta"]["version"]
)
# And ETag should be the same (indicating cache hit)
self.assertEqual(response1["ETag"], response2["ETag"])
class GroupsIntegrationTest(TestCase):

6
app/groups/views.py
View file

@ -1,4 +1,3 @@
import hashlib
from rest_framework import status
from rest_framework.views import APIView
from rest_framework.response import Response
@ -128,10 +127,6 @@ class GroupMessagesView(APIView):
member_profiles = Profile.objects.filter(posts__group=group_slug).distinct()
members_list = [profile.feed for profile in member_profiles]
# Generate version hash
version_string = "".join(sorted([p["post"] for p in messages_tree]))
version = hashlib.sha256(version_string.encode()).hexdigest()[:8]
# URL encode the group_slug for the join link template
from urllib.parse import quote
@ -144,7 +139,6 @@ class GroupMessagesView(APIView):
"meta": {
"group": group_display_name,
"members": members_list,
"version": version,
},
"_links": {
"self": {"href": f"/groups/{encoded_group_slug}/", "method": "GET"},

0
app/interactions/__init__.py Normal file
View file

6
app/interactions/apps.py Normal file
View file

@ -0,0 +1,6 @@
from django.apps import AppConfig
class InteractionsConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "app.interactions"

323
app/interactions/tests.py Normal file
View file

@ -0,0 +1,323 @@
from django.test import TestCase
from rest_framework.test import APIClient
from rest_framework import status
from app.feeds.models import Profile, Post
class InteractionsViewTest(TestCase):
"""Test cases for the InteractionsView API using Given/When/Then structure."""
def setUp(self):
self.client = APIClient()
self.interactions_url = "/interactions/"
# Create test profiles
self.profile1 = Profile.objects.create(
feed="https://example.com/social.org",
title="Example Profile",
nick="example_user",
description="Test profile 1",
)
self.profile2 = Profile.objects.create(
feed="https://alice.com/social.org",
title="Alice Profile",
nick="alice",
description="Alice's profile",
)
self.profile3 = Profile.objects.create(
feed="https://bob.com/social.org",
title="Bob Profile",
nick="bob",
description="Bob's profile",
)
self.profile4 = Profile.objects.create(
feed="https://charlie.com/social.org",
title="Charlie Profile",
nick="charlie",
description="Charlie's profile",
)
# Create original post
self.original_post = Post.objects.create(
profile=self.profile1,
post_id="2025-02-05T10:00:00+0100",
content="This is an amazing discovery!",
)
# Create reactions
self.reaction1 = Post.objects.create(
profile=self.profile2,
post_id="2025-02-05T13:15:00+0100",
content="",
mood="❤",
reply_to=f"{self.profile1.feed}#{self.original_post.post_id}",
)
self.reaction2 = Post.objects.create(
profile=self.profile3,
post_id="2025-02-05T14:30:00+0100",
content="",
mood="🚀",
reply_to=f"{self.profile1.feed}#{self.original_post.post_id}",
)
# Create replies
self.reply1 = Post.objects.create(
profile=self.profile4,
post_id="2025-02-05T12:30:00+0100",
content="Great post!",
mood="",
reply_to=f"{self.profile1.feed}#{self.original_post.post_id}",
)
self.reply2 = Post.objects.create(
profile=self.profile2,
post_id="2025-02-05T15:00:00+0100",
content="I agree!",
mood="",
reply_to=f"{self.profile1.feed}#{self.original_post.post_id}",
)
# Create boosts
self.boost1 = Post.objects.create(
profile=self.profile2,
post_id="2025-02-05T14:00:00+0100",
content="Guys, you have to see this!",
include=f"{self.profile1.feed}#{self.original_post.post_id}",
)
self.boost2 = Post.objects.create(
profile=self.profile3,
post_id="2025-02-05T15:30:00+0100",
content="",
include=f"{self.profile1.feed}#{self.original_post.post_id}",
)
def test_get_interactions_success(self):
"""Test GET /interactions/?post=<post_url> returns all interactions."""
# Given: A post with reactions, replies, and boosts
post_url = f"{self.profile1.feed}#{self.original_post.post_id}"
# When: We request interactions for the post
response = self.client.get(self.interactions_url, {"post": post_url})
# Then: We should get all interactions successfully
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["type"], "Success")
self.assertEqual(response.data["errors"], [])
# Then: Response should contain all three types
data = response.data["data"]
self.assertIn("reactions", data)
self.assertIn("replies", data)
self.assertIn("boosts", data)
# Verify reactions
reactions = data["reactions"]
self.assertEqual(len(reactions), 2)
self.assertEqual(reactions[0]["emoji"], "🚀") # Most recent first
self.assertEqual(reactions[1]["emoji"], "❤")
# Verify replies
replies = data["replies"]
self.assertEqual(len(replies), 2)
self.assertIn(f"{self.profile2.feed}#{self.reply2.post_id}", replies)
self.assertIn(f"{self.profile4.feed}#{self.reply1.post_id}", replies)
# Verify boosts
boosts = data["boosts"]
self.assertEqual(len(boosts), 2)
self.assertIn(f"{self.profile2.feed}#{self.boost1.post_id}", boosts)
self.assertIn(f"{self.profile3.feed}#{self.boost2.post_id}", boosts)
# Verify meta
meta = response.data["meta"]
self.assertEqual(meta["post"], post_url)
self.assertEqual(meta["total_reactions"], 2)
self.assertEqual(meta["total_replies"], 2)
self.assertEqual(meta["total_boosts"], 2)
self.assertEqual(meta["parentChain"], []) # Original post has no parents
# Verify links
links = response.data["_links"]
self.assertIn("self", links)
self.assertIn("reactions", links)
self.assertIn("replies", links)
self.assertIn("boosts", links)
def test_get_interactions_with_parent_chain(self):
"""Test GET /interactions/ includes parent chain for reply posts."""
# Given: A post that is a reply (has parents)
parent_post = Post.objects.create(
profile=self.profile2,
post_id="2025-02-05T08:00:00+0100",
content="Parent post",
)
child_post = Post.objects.create(
profile=self.profile1,
post_id="2025-02-05T09:00:00+0100",
content="Reply to parent",
reply_to=f"{self.profile2.feed}#{parent_post.post_id}",
)
post_url = f"{self.profile1.feed}#{child_post.post_id}"
# When: We request interactions for the child post
response = self.client.get(self.interactions_url, {"post": post_url})
# Then: Should include parent chain
self.assertEqual(response.status_code, status.HTTP_200_OK)
meta = response.data["meta"]
self.assertIn("parentChain", meta)
self.assertEqual(len(meta["parentChain"]), 1)
self.assertEqual(
meta["parentChain"][0], f"{self.profile2.feed}#{parent_post.post_id}"
)
def test_get_interactions_no_interactions(self):
"""Test GET /interactions/ when post has no interactions."""
# Given: A post without any interactions
lonely_post = Post.objects.create(
profile=self.profile1,
post_id="2025-02-05T20:00:00+0100",
content="Lonely post",
)
post_url = f"{self.profile1.feed}#{lonely_post.post_id}"
# When: We request interactions
response = self.client.get(self.interactions_url, {"post": post_url})
# Then: Should return empty arrays
self.assertEqual(response.status_code, status.HTTP_200_OK)
data = response.data["data"]
self.assertEqual(len(data["reactions"]), 0)
self.assertEqual(len(data["replies"]), 0)
self.assertEqual(len(data["boosts"]), 0)
meta = response.data["meta"]
self.assertEqual(meta["total_reactions"], 0)
self.assertEqual(meta["total_replies"], 0)
self.assertEqual(meta["total_boosts"], 0)
def test_get_interactions_missing_post_parameter(self):
"""Test GET /interactions/ without post parameter returns error."""
# Given: No post parameter provided
# When: We request interactions without post parameter
response = self.client.get(self.interactions_url)
# Then: We should get a 400 error
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(response.data["type"], "Error")
self.assertIn("'post' parameter is required", response.data["errors"])
def test_get_interactions_invalid_post_url_format(self):
"""Test GET /interactions/?post=<invalid_url> returns error."""
# Given: An invalid post URL (missing #)
invalid_url = "https://example.com/social.org"
# When: We request interactions with invalid URL
response = self.client.get(self.interactions_url, {"post": invalid_url})
# Then: We should get a 400 error
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(response.data["type"], "Error")
self.assertIn("Invalid post URL format", response.data["errors"][0])
def test_get_interactions_post_not_found(self):
"""Test GET /interactions/?post=<nonexistent_post> returns 404."""
# Given: A non-existent post URL
post_url = "https://nonexistent.com/social.org#2025-01-01T00:00:00+00:00"
# When: We request interactions for non-existent post
response = self.client.get(self.interactions_url, {"post": post_url})
# Then: We should get a 404 error
self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)
self.assertEqual(response.data["type"], "Error")
self.assertIn("Post not found", response.data["errors"])
def test_interactions_excludes_poll_votes(self):
"""Test that poll votes are excluded from reactions and replies."""
# Given: A post with a poll vote (should be excluded)
poll_post = Post.objects.create(
profile=self.profile2,
post_id="2025-02-05T16:00:00+0100",
content="Poll post",
)
# Create a poll vote (reply with poll_votes relationship)
vote_post = Post.objects.create(
profile=self.profile3,
post_id="2025-02-05T17:00:00+0100",
content="",
reply_to=f"{self.profile1.feed}#{self.original_post.post_id}",
)
from app.feeds.models import PollVote
PollVote.objects.create(
post=vote_post, poll_post=poll_post, poll_option="Option A"
)
post_url = f"{self.profile1.feed}#{self.original_post.post_id}"
# When: We request interactions
response = self.client.get(self.interactions_url, {"post": post_url})
# Then: Poll vote should not be in replies or reactions
self.assertEqual(response.status_code, status.HTTP_200_OK)
vote_url = f"{self.profile3.feed}#{vote_post.post_id}"
data = response.data["data"]
self.assertNotIn(vote_url, data["replies"])
for reaction in data["reactions"]:
self.assertNotEqual(reaction["post"], vote_url)
def test_interactions_caching(self):
"""Test that interactions responses are cached."""
# Given: A post with interactions
post_url = f"{self.profile1.feed}#{self.original_post.post_id}"
# When: We request interactions twice
response1 = self.client.get(self.interactions_url, {"post": post_url})
response2 = self.client.get(self.interactions_url, {"post": post_url})
# Then: Both responses should be identical
self.assertEqual(response1.data, response2.data)
self.assertEqual(response1.status_code, status.HTTP_200_OK)
self.assertEqual(response2.status_code, status.HTTP_200_OK)
def test_interactions_ordered_by_recency(self):
"""Test that all interactions are ordered from most recent to oldest."""
# Given: A post with interactions
post_url = f"{self.profile1.feed}#{self.original_post.post_id}"
# When: We request interactions
response = self.client.get(self.interactions_url, {"post": post_url})
# Then: Items should be ordered by post_id descending (most recent first)
data = response.data["data"]
# Check reactions order
if len(data["reactions"]) > 1:
for i in range(len(data["reactions"]) - 1):
current_id = data["reactions"][i]["post"].split("#")[1]
next_id = data["reactions"][i + 1]["post"].split("#")[1]
self.assertGreater(current_id, next_id)
# Check replies order
if len(data["replies"]) > 1:
for i in range(len(data["replies"]) - 1):
current_id = data["replies"][i].split("#")[1]
next_id = data["replies"][i + 1].split("#")[1]
self.assertGreater(current_id, next_id)
# Check boosts order
if len(data["boosts"]) > 1:
for i in range(len(data["boosts"]) - 1):
current_id = data["boosts"][i].split("#")[1]
next_id = data["boosts"][i + 1].split("#")[1]
self.assertGreater(current_id, next_id)

8
app/interactions/urls.py Normal file
View file

@ -0,0 +1,8 @@
from django.urls import path
from .views import InteractionsView
app_name = "interactions"
urlpatterns = [
path("", InteractionsView.as_view(), name="interactions"),
]

134
app/interactions/views.py Normal file
View file

@ -0,0 +1,134 @@
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework import status
from django.core.cache import cache
import logging
from app.feeds.models import Post, Profile
from app.feeds.utils import get_parent_chain
from app.reactions.utils import get_reactions_for_post
from app.replies.utils import get_direct_replies_for_post
from app.boosts.utils import get_boosts_for_post
logger = logging.getLogger(__name__)
class InteractionsView(APIView):
"""Get all interactions (reactions, replies, boosts) for a specific post"""
def get(self, request):
post_url = request.query_params.get("post")
if not post_url:
return Response(
{
"type": "Error",
"errors": ["'post' parameter is required"],
"data": None,
},
status=status.HTTP_400_BAD_REQUEST,
)
# Parse the post URL format: https://feed.com/social.org#post_id
try:
if "#" not in post_url:
raise ValueError("Invalid post URL format")
feed_url, post_id = post_url.split("#", 1)
except ValueError:
return Response(
{
"type": "Error",
"errors": ["Invalid post URL format. Expected: feed_url#post_id"],
"data": None,
},
status=status.HTTP_400_BAD_REQUEST,
)
cache_key = f"interactions_{feed_url}_{post_id}"
cached_response = cache.get(cache_key)
if cached_response is not None:
return Response(cached_response, status=status.HTTP_200_OK)
# Find the original post
try:
profile = Profile.objects.get(feed=feed_url)
original_post = Post.objects.get(profile=profile, post_id=post_id)
except (Profile.DoesNotExist, Post.DoesNotExist):
return Response(
{
"type": "Error",
"errors": ["Post not found"],
"data": None,
},
status=status.HTTP_404_NOT_FOUND,
)
original_post_url = f"{feed_url}#{post_id}"
# Get reactions using shared utility function
reactions_qs = get_reactions_for_post(original_post_url)
reactions_data = [
{
"post": f"{reaction.profile.feed}#{reaction.post_id}",
"emoji": reaction.mood,
}
for reaction in reactions_qs
]
# Get replies using shared utility function
replies_qs = get_direct_replies_for_post(original_post_url)
replies_data = [f"{reply.profile.feed}#{reply.post_id}" for reply in replies_qs]
# Get boosts using shared utility function
boosts_qs = get_boosts_for_post(original_post_url)
boosts_data = [f"{boost.profile.feed}#{boost.post_id}" for boost in boosts_qs]
# Get parent chain
parent_chain = get_parent_chain(original_post)
# URL encode the post_url for the self link
from urllib.parse import quote
encoded_post_url = quote(post_url, safe="")
encoded_feed_url = quote(feed_url, safe="")
response_data = {
"type": "Success",
"errors": [],
"data": {
"reactions": reactions_data,
"replies": replies_data,
"boosts": boosts_data,
},
"meta": {
"post": original_post_url,
"total_reactions": len(reactions_data),
"total_replies": len(replies_data),
"total_boosts": len(boosts_data),
"parentChain": parent_chain,
},
"_links": {
"self": {
"href": f"/interactions/?post={encoded_post_url}",
"method": "GET",
},
"reactions": {
"href": f"/reactions/?feed={encoded_feed_url}",
"method": "GET",
},
"replies": {
"href": f"/replies/?post={encoded_post_url}",
"method": "GET",
},
"boosts": {
"href": f"/boosts/?post={encoded_post_url}",
"method": "GET",
},
},
}
# Cache permanently (will be cleared by scan_feeds task)
cache.set(cache_key, response_data, None)
return Response(response_data, status=status.HTTP_200_OK)

10
app/mentions/tests.py
View file

@ -92,7 +92,10 @@ class MentionsViewTest(TestCase):
meta = response.data["meta"]
self.assertEqual(meta["feed"], feed_url)
self.assertEqual(meta["total"], 2)
self.assertIn("version", meta)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_get_mentions_no_mentions(self):
"""Test GET /mentions/ returns empty array for profile with no mentions."""
@ -112,7 +115,10 @@ class MentionsViewTest(TestCase):
meta = response.data["meta"]
self.assertEqual(meta["feed"], feed_url)
self.assertEqual(meta["total"], 0)
self.assertIn("version", meta)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_get_mentions_nonexistent_profile(self):
"""Test GET /mentions/ returns 404 for nonexistent profile."""

7
app/mentions/views.py
View file

@ -3,7 +3,6 @@ from rest_framework.response import Response
from rest_framework import status
from django.core.cache import cache
import logging
import hashlib
from app.feeds.models import Profile, Mention
@ -61,10 +60,6 @@ class MentionsView(APIView):
post_url = f"{mention.post.profile.feed}#{mention.post.post_id}"
mentions_data.append(post_url)
# Generate version hash based on profile's last update and mentions count
version_string = f"{profile.last_updated.isoformat()}_{len(mentions_data)}"
version = hashlib.md5(version_string.encode()).hexdigest()[:8]
# URL encode the feed_url for the self link
from urllib.parse import quote
@ -74,7 +69,7 @@ class MentionsView(APIView):
"type": "Success",
"errors": [],
"data": mentions_data,
"meta": {"feed": feed_url, "total": len(mentions_data), "version": version},
"meta": {"feed": feed_url, "total": len(mentions_data)},
"_links": {
"self": {"href": f"/mentions/?feed={encoded_feed_url}", "method": "GET"}
},

78
app/notifications/tests.py
View file

@ -104,7 +104,10 @@ class NotificationsViewTest(TestCase):
self.assertEqual(meta["by_type"]["mentions"], 1)
self.assertEqual(meta["by_type"]["reactions"], 1)
self.assertEqual(meta["by_type"]["replies"], 1)
self.assertIn("version", meta)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_get_notifications_filtered_by_mention(self):
"""Test GET /notifications/?feed=<feed_url>&type=mention returns only mentions."""
@ -343,3 +346,76 @@ class NotificationsViewTest(TestCase):
self.assertIn("post", reply)
self.assertIn("parent", reply)
self.assertNotIn("emoji", reply)
def test_get_notifications_with_boosts(self):
"""Test GET /notifications/?feed=<feed_url> includes boosts."""
# Given: A profile with posts that have been boosted
boost_post = Post.objects.create(
profile=self.profile2,
post_id="2025-01-01T17:00:00+00:00",
content="Boosting this amazing post!",
include=f"{self.profile1.feed}#{self.post1.post_id}",
)
# When: We request all notifications for the profile
response = self.client.get(self.notifications_url, {"feed": self.profile1.feed})
# Then: We should get notifications including the boost
self.assertEqual(response.status_code, status.HTTP_200_OK)
data = response.data["data"]
# Find the boost notification
boost_notifications = [n for n in data if n["type"] == "boost"]
self.assertEqual(len(boost_notifications), 1)
boost_notif = boost_notifications[0]
self.assertEqual(boost_notif["type"], "boost")
self.assertEqual(
boost_notif["post"], f"{self.profile2.feed}#{boost_post.post_id}"
)
self.assertEqual(
boost_notif["boosted"], f"{self.profile1.feed}#{self.post1.post_id}"
)
# Then: Meta should include boost count
meta = response.data["meta"]
self.assertEqual(meta["by_type"]["boosts"], 1)
self.assertEqual(meta["total"], 4) # 1 mention + 1 reaction + 1 reply + 1 boost
def test_get_notifications_filtered_by_boost(self):
"""Test GET /notifications/?feed=<feed_url>&type=boost returns only boosts."""
# Given: A profile with all notification types including boosts
Post.objects.create(
profile=self.profile2,
post_id="2025-01-01T17:00:00+00:00",
content="Boosting!",
include=f"{self.profile1.feed}#{self.post1.post_id}",
)
Post.objects.create(
profile=self.profile3,
post_id="2025-01-01T18:00:00+00:00",
content="",
include=f"{self.profile1.feed}#{self.post2.post_id}",
)
# When: We request only boosts
response = self.client.get(
self.notifications_url, {"feed": self.profile1.feed, "type": "boost"}
)
# Then: Should only return boosts
self.assertEqual(response.status_code, status.HTTP_200_OK)
data = response.data["data"]
self.assertEqual(len(data), 2)
for boost in data:
self.assertEqual(boost["type"], "boost")
self.assertIn("post", boost)
self.assertIn("boosted", boost)
self.assertNotIn("emoji", boost)
self.assertNotIn("parent", boost)
# Then: Meta should show filtered results
meta = response.data["meta"]
self.assertEqual(meta["total"], 2)
self.assertEqual(meta["by_type"]["boosts"], 2)

31
app/notifications/views.py
View file

@ -4,7 +4,6 @@ from rest_framework import status
from django.core.cache import cache
from django.db.models import Q
import logging
import hashlib
from app.feeds.models import Profile, Post, Mention
@ -31,7 +30,7 @@ class NotificationsView(APIView):
feed_url = feed_url.strip()
# Validate type parameter if provided
valid_types = ["mention", "reaction", "reply"]
valid_types = ["mention", "reaction", "reply", "boost"]
if notification_type and notification_type not in valid_types:
return Response(
{
@ -65,7 +64,7 @@ class NotificationsView(APIView):
)
notifications_data = []
counts = {"mentions": 0, "reactions": 0, "replies": 0}
counts = {"mentions": 0, "reactions": 0, "replies": 0, "boosts": 0}
# Get all post IDs from this profile (for reactions and replies)
profile_post_ids = list(profile.posts.values_list("post_id", flat=True))
@ -128,6 +127,27 @@ class NotificationsView(APIView):
)
counts["replies"] = len(replies)
# 4. Get boosts (if not filtering or filtering for boosts)
# Build include patterns (feed#post_id) for all posts from this profile
include_patterns = [f"{feed_url}#{post_id}" for post_id in profile_post_ids]
if not notification_type or notification_type == "boost":
boosts = (
Post.objects.filter(include__in=include_patterns)
.select_related("profile")
.order_by("-post_id")
)
for boost in boosts:
notifications_data.append(
{
"type": "boost",
"post": f"{boost.profile.feed}#{boost.post_id}",
"boosted": boost.include,
"_sort_key": boost.post_id,
}
)
counts["boosts"] = len(boosts)
# Sort all notifications by post_id (most recent first)
notifications_data.sort(key=lambda x: x["_sort_key"], reverse=True)
@ -135,10 +155,6 @@ class NotificationsView(APIView):
for notification in notifications_data:
del notification["_sort_key"]
# Generate version hash based on profile's last update and notification counts
version_string = f"{profile.last_updated.isoformat()}_{sum(counts.values())}"
version = hashlib.md5(version_string.encode()).hexdigest()[:8]
# URL encode the feed_url for links
from urllib.parse import quote
@ -152,7 +168,6 @@ class NotificationsView(APIView):
"feed": feed_url,
"total": len(notifications_data),
"by_type": counts,
"version": version,
},
"_links": {
"self": {

17
app/polls/tests.py
View file

@ -98,10 +98,13 @@ class PollsViewTest(TestCase):
self.assertIsInstance(poll_url, str)
self.assertIn("#", poll_url) # Should contain the # separator
# Then: Meta should contain total and version
# Then: Meta should contain total
self.assertIn("meta", response.data)
self.assertEqual(response.data["meta"]["total"], 2)
self.assertIn("version", response.data["meta"])
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_get_polls_for_specific_feed(self):
"""Test GET /polls?feed=<url> returns polls for specific feed."""
@ -120,7 +123,10 @@ class PollsViewTest(TestCase):
# Then: Response should contain feed metadata
self.assertEqual(response.data["meta"]["feed"], self.profile1.feed)
self.assertEqual(response.data["meta"]["total"], 2)
self.assertIn("version", response.data["meta"])
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_get_polls_for_nonexistent_feed(self):
"""Test GET /polls?feed=<nonexistent> returns 404."""
@ -336,7 +342,10 @@ class PollVotesViewTest(TestCase):
meta = response.data["meta"]
self.assertEqual(meta["poll"], post_url)
self.assertEqual(meta["total_votes"], 2)
self.assertIn("version", meta)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_get_poll_votes_nonexistent_poll(self):
"""Test GET /polls/votes/ returns 404 for nonexistent poll."""

22
app/polls/views.py
View file

@ -4,7 +4,6 @@ from rest_framework import status
from django.core.cache import cache
from django.utils import timezone
import logging
import hashlib
from app.feeds.models import Post, Profile, PollVote
@ -37,7 +36,6 @@ class PollsView(APIView):
)
# Get all polls (both active and expired)
current_time = timezone.now()
polls = (
Post.objects.filter(poll_end__isnull=False)
.select_related("profile")
@ -51,17 +49,12 @@ class PollsView(APIView):
poll_url = f"{poll.profile.feed}#{poll.post_id}"
polls_data.append(poll_url)
# Generate version hash for polls
version_string = f"all_polls_{len(polls_data)}_{current_time.isoformat()}"
version = hashlib.md5(version_string.encode()).hexdigest()[:8]
response_data = {
"type": "Success",
"errors": [],
"data": polls_data,
"meta": {
"total": len(polls_data),
"version": version,
},
"_links": {
"self": {"href": "/polls/", "method": "GET"},
@ -123,10 +116,6 @@ class PollsView(APIView):
}
polls_data.append(poll_data)
# Generate version hash
version_string = f"{profile.last_updated.isoformat()}_{len(polls_data)}"
version = hashlib.md5(version_string.encode()).hexdigest()[:8]
response_data = {
"type": "Success",
"errors": [],
@ -134,7 +123,6 @@ class PollsView(APIView):
"meta": {
"feed": feed_url,
"total": len(polls_data),
"version": version,
},
}
@ -184,10 +172,6 @@ class PollsView(APIView):
}
votes_data.append(vote_data)
# Generate version hash
version_string = f"{voter_profile.last_updated.isoformat()}_{len(votes_data)}"
version = hashlib.md5(version_string.encode()).hexdigest()[:8]
response_data = {
"type": "Success",
"errors": [],
@ -195,7 +179,6 @@ class PollsView(APIView):
"meta": {
"voter": voter_url,
"total": len(votes_data),
"version": version,
},
}
@ -296,10 +279,6 @@ class PollVotesView(APIView):
total_votes += len(option_votes)
data.append({"option": option, "votes": option_votes})
# Generate version hash
version_string = f"{poll_post.updated_at.isoformat()}_{total_votes}"
version = hashlib.md5(version_string.encode()).hexdigest()[:8]
# URL encode the post_url for the self link
from urllib.parse import quote
@ -312,7 +291,6 @@ class PollVotesView(APIView):
"meta": {
"poll": f"{poll_feed}#{poll_id}",
"total_votes": total_votes,
"version": version,
},
"_links": {
"self": {

0
app/profile/__init__.py Normal file
View file

6
app/profile/apps.py Normal file
View file

@ -0,0 +1,6 @@
from django.apps import AppConfig
class ProfileConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "app.profile"

176
app/profile/tests.py Normal file
View file

@ -0,0 +1,176 @@
from django.test import TestCase
from rest_framework.test import APIClient
from rest_framework import status
from app.feeds.models import Follow, Profile
class ProfileViewTest(TestCase):
"""Test cases for the ProfileView API using Given/When/Then structure."""
def setUp(self):
self.client = APIClient()
self.url = "/profile/"
self.profile_main = Profile.objects.create(
feed="https://example.com/social.org",
title="Example Profile",
nick="example_user",
)
self.profile_alice = Profile.objects.create(
feed="https://alice.org/social.org",
title="Alice",
nick="alice",
)
self.profile_bob = Profile.objects.create(
feed="https://bob.org/social.org",
title="Bob",
nick="bob",
)
self.profile_carol = Profile.objects.create(
feed="https://carol.org/social.org",
title="Carol",
nick="carol",
)
Follow.objects.create(follower=self.profile_alice, followed=self.profile_main)
Follow.objects.create(follower=self.profile_bob, followed=self.profile_main)
def test_get_profile_with_followers(self):
"""Test GET /profile/?feed=<url> returns correct followers list."""
# Given: A profile followed by two users
feed_url = self.profile_main.feed
# When: We request the profile
response = self.client.get(self.url, {"feed": feed_url})
# Then: Response is successful
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["type"], "Success")
self.assertEqual(response.data["errors"], [])
# Then: Data contains feed and followers
data = response.data["data"]
self.assertEqual(data["feed"], feed_url)
self.assertIsInstance(data["followers"], list)
self.assertIn(self.profile_alice.feed, data["followers"])
self.assertIn(self.profile_bob.feed, data["followers"])
self.assertNotIn(self.profile_carol.feed, data["followers"])
# Then: Meta contains correct counts
meta = response.data["meta"]
self.assertEqual(meta["feed"], feed_url)
self.assertEqual(meta["total_followers"], 2)
def test_get_profile_no_followers(self):
"""Test GET /profile/ returns empty followers for profile with no followers."""
# Given: A profile with no followers
feed_url = self.profile_carol.feed
# When: We request the profile
response = self.client.get(self.url, {"feed": feed_url})
# Then: Response is successful with empty followers
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["type"], "Success")
self.assertEqual(response.data["data"]["followers"], [])
self.assertEqual(response.data["meta"]["total_followers"], 0)
def test_get_profile_not_found(self):
"""Test GET /profile/ returns 404 for non-existent feed."""
# Given: A feed URL not registered in the relay
nonexistent_feed = "https://nonexistent.com/social.org"
# When: We request the profile
response = self.client.get(self.url, {"feed": nonexistent_feed})
# Then: We get 404
self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)
self.assertEqual(response.data["type"], "Error")
self.assertIn("Profile not found", response.data["errors"][0])
self.assertIsNone(response.data["data"])
def test_get_profile_missing_feed_param(self):
"""Test GET /profile/ returns 400 when feed param is missing."""
# When: We request without the feed parameter
response = self.client.get(self.url)
# Then: We get 400
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(response.data["type"], "Error")
self.assertIn("required", response.data["errors"][0])
def test_get_profile_response_format(self):
"""Test GET /profile/ response matches README specification."""
# Given: A registered profile
feed_url = self.profile_main.feed
# When: We request the profile
response = self.client.get(self.url, {"feed": feed_url})
# Then: Response keys are correct
self.assertIn("type", response.data)
self.assertIn("errors", response.data)
self.assertIn("data", response.data)
self.assertIn("meta", response.data)
self.assertIn("_links", response.data)
data = response.data["data"]
self.assertIn("feed", data)
self.assertIn("followers", data)
meta = response.data["meta"]
self.assertIn("feed", meta)
self.assertIn("total_followers", meta)
links = response.data["_links"]
self.assertIn("self", links)
self.assertIn("href", links["self"])
self.assertIn("method", links["self"])
self.assertEqual(links["self"]["method"], "GET")
def test_get_profile_self_link_encoded(self):
"""Test that the self link URL-encodes the feed parameter."""
# Given: A feed URL with special characters
feed_url = self.profile_main.feed
# When: We request the profile
response = self.client.get(self.url, {"feed": feed_url})
# Then: Self link is URL-encoded
self.assertEqual(response.status_code, status.HTTP_200_OK)
href = response.data["_links"]["self"]["href"]
self.assertIn("%3A", href) # : is encoded
self.assertIn("%2F", href) # / is encoded
def test_get_profile_http_methods(self):
"""Test that only GET is allowed on the profile endpoint."""
# Given: A valid feed URL
params = {"feed": self.profile_main.feed}
# When/Then: Non-GET methods return 405
self.assertEqual(
self.client.post(self.url, params).status_code,
status.HTTP_405_METHOD_NOT_ALLOWED,
)
self.assertEqual(
self.client.put(self.url, params).status_code,
status.HTTP_405_METHOD_NOT_ALLOWED,
)
self.assertEqual(
self.client.delete(self.url, params).status_code,
status.HTTP_405_METHOD_NOT_ALLOWED,
)
def test_get_profile_followers_are_feeds_only(self):
"""Test that followers list contains only feed URLs, not profile metadata."""
# Given: A profile with a follower
feed_url = self.profile_main.feed
# When: We request the profile
response = self.client.get(self.url, {"feed": feed_url})
# Then: Each follower entry is a plain URL string
for follower in response.data["data"]["followers"]:
self.assertIsInstance(follower, str)
self.assertTrue(follower.startswith("http"))

7
app/profile/urls.py Normal file
View file

@ -0,0 +1,7 @@
from django.urls import path
from app.profile.views import ProfileView
urlpatterns = [
path("", ProfileView.as_view(), name="profile"),
]

76
app/profile/views.py Normal file
View file

@ -0,0 +1,76 @@
from urllib.parse import quote
from rest_framework import status
from rest_framework.response import Response
from rest_framework.views import APIView
from django.core.cache import cache
from app.feeds.models import Follow, Profile
class ProfileView(APIView):
"""Get profile information including followers for a given feed URL."""
def get(self, request):
feed_url = request.query_params.get("feed")
if not feed_url or not feed_url.strip():
return Response(
{
"type": "Error",
"errors": ["Feed URL parameter is required"],
"data": None,
},
status=status.HTTP_400_BAD_REQUEST,
)
feed_url = feed_url.strip()
cache_key = f"profile_{feed_url}"
cached_response = cache.get(cache_key)
if cached_response is not None:
return Response(cached_response, status=status.HTTP_200_OK)
try:
profile = Profile.objects.get(feed=feed_url)
except Profile.DoesNotExist:
return Response(
{
"type": "Error",
"errors": ["Profile not found for the given feed URL"],
"data": None,
},
status=status.HTTP_404_NOT_FOUND,
)
followers = (
Follow.objects.filter(followed=profile)
.select_related("follower")
.order_by("follower__feed")
)
follower_feeds = [f.follower.feed for f in followers]
encoded_feed_url = quote(feed_url, safe="")
response_data = {
"type": "Success",
"errors": [],
"data": {
"feed": feed_url,
"followers": follower_feeds,
},
"meta": {
"feed": feed_url,
"total_followers": len(follower_feeds),
},
"_links": {
"self": {
"href": f"/profile/?feed={encoded_feed_url}",
"method": "GET",
}
},
}
cache.set(cache_key, response_data, None)
return Response(response_data, status=status.HTTP_200_OK)

116
app/public/tests.py Normal file
View file

@ -0,0 +1,116 @@
from django.test import TestCase
from rest_framework.test import APIClient
from rest_framework import status
class RootViewTest(TestCase):
"""Test cases for the root endpoint."""
def setUp(self):
self.client = APIClient()
self.root_url = "/"
def test_root_endpoint_success(self):
"""Test GET / returns success response with HATEOAS links."""
# Given: The root endpoint
# When: We request the root endpoint
response = self.client.get(self.root_url)
# Then: We should get success response
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.json()["type"], "Success")
self.assertEqual(response.json()["errors"], [])
# Then: Should have data with name and description
data = response.json()["data"]
self.assertIn("name", data)
self.assertIn("description", data)
self.assertEqual(data["name"], "Org Social Relay")
def test_root_endpoint_hateoas_links(self):
"""Test GET / returns all expected HATEOAS links."""
# Given: The root endpoint
# When: We request the root endpoint
response = self.client.get(self.root_url)
# Then: Should have _links with all endpoints
links = response.json()["_links"]
expected_links = [
"self",
"feeds",
"add-feed",
"mentions",
"replies",
"notifications",
"reactions",
"replies-to",
"search",
"groups",
"group-messages",
"join-group",
"polls",
"poll-votes",
]
for link_name in expected_links:
self.assertIn(link_name, links, f"Missing link: {link_name}")
def test_root_endpoint_has_caching_headers(self):
"""Test GET / returns ETag and Last-Modified headers."""
# Given: The root endpoint
# When: We request the root endpoint
response = self.client.get(self.root_url)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
# Then: ETag should be properly formatted (quoted)
etag = response["ETag"]
self.assertTrue(etag.startswith('"') and etag.endswith('"'))
def test_root_endpoint_consistent_etag(self):
"""Test GET / returns consistent ETag for same content."""
# Given: The root endpoint
# When: We request the root endpoint multiple times
response1 = self.client.get(self.root_url)
response2 = self.client.get(self.root_url)
# Then: ETag should be the same (content is static)
self.assertEqual(response1["ETag"], response2["ETag"])
def test_root_endpoint_response_format_compliance(self):
"""Test root endpoint response format compliance."""
# Given: The root endpoint
# When: We request the root endpoint
response = self.client.get(self.root_url)
# Then: Response should match expected format
self.assertEqual(response.status_code, status.HTTP_200_OK)
response_data = response.json()
self.assertIn("type", response_data)
self.assertIn("errors", response_data)
self.assertIn("data", response_data)
self.assertIn("_links", response_data)
self.assertIsInstance(response_data["errors"], list)
self.assertIsInstance(response_data["data"], dict)
self.assertIsInstance(response_data["_links"], dict)
def test_root_endpoint_all_methods_return_same_content(self):
"""Test that root endpoint returns same content for all HTTP methods."""
# Given: The root endpoint
# When: We try different HTTP methods
get_response = self.client.get(self.root_url)
post_response = self.client.post(self.root_url)
# Then: All methods should return 200 with same content
self.assertEqual(get_response.status_code, status.HTTP_200_OK)
self.assertEqual(post_response.status_code, status.HTTP_200_OK)
self.assertEqual(get_response.json()["type"], "Success")
self.assertEqual(post_response.json()["type"], "Success")

48
app/public/views.py
View file

@ -15,6 +15,11 @@ def root_view(request):
"self": {"href": "/", "method": "GET"},
"feeds": {"href": "/feeds/", "method": "GET"},
"add-feed": {"href": "/feeds/", "method": "POST"},
"feed-content": {
"href": "/feed-content/?feed={feed_url}",
"method": "GET",
"templated": True,
},
"mentions": {
"href": "/mentions/?feed={feed_url}",
"method": "GET",
@ -30,6 +35,15 @@ def root_view(request):
"method": "GET",
"templated": True,
},
"sse-notifications": {
"href": "/sse/notifications/?feed={feed_url}",
"method": "GET",
"templated": True,
},
"sse-notifications-all": {
"href": "/sse/notifications/",
"method": "GET",
},
"reactions": {
"href": "/reactions/?feed={feed_url}",
"method": "GET",
@ -40,6 +54,16 @@ def root_view(request):
"method": "GET",
"templated": True,
},
"boosts": {
"href": "/boosts/?post={post_url}",
"method": "GET",
"templated": True,
},
"interactions": {
"href": "/interactions/?post={post_url}",
"method": "GET",
"templated": True,
},
"search": {
"href": "/search/?q={query}",
"method": "GET",
@ -62,6 +86,30 @@ def root_view(request):
"method": "GET",
"templated": True,
},
"profile": {
"href": "/profile/?feed={feed_url}",
"method": "GET",
"templated": True,
},
"rss": {
"href": "/rss.xml",
"method": "GET",
"description": "RSS feed of latest posts (supports ?tag={tag} and ?feed={feed_url} filters)",
},
"stats": {"href": "/stats/", "method": "GET"},
"bridge": {"href": "/bridge/", "method": "GET"},
"bridge-activitypub": {
"href": "/bridge/activitypub/@{user}@{instance}/",
"method": "GET",
"templated": True,
"description": "ActivityPub account as a virtual social.org feed",
},
"bridge-rss": {
"href": "/bridge/rss/?url={feed_url}",
"method": "GET",
"templated": True,
"description": "RSS/Atom feed as a virtual social.org feed",
},
},
}
)

27
app/reactions/renderers.py Normal file
View file

@ -0,0 +1,27 @@
from rest_framework.renderers import JSONRenderer
import json
class UTF8JSONRenderer(JSONRenderer):
"""
JSON renderer that preserves UTF-8 characters (like emojis) instead of escaping them.
"""
charset = "utf-8"
def render(self, data, accepted_media_type=None, renderer_context=None):
if data is None:
return bytes()
renderer_context = renderer_context or {}
indent = self.get_indent(accepted_media_type, renderer_context)
if indent is None:
separators = (",", ":")
else:
separators = (",", ": ")
ret = json.dumps(data, ensure_ascii=False, indent=indent, separators=separators)
# Handle invalid surrogate characters by using 'surrogatepass' error handler
return ret.encode("utf-8", errors="surrogatepass")

10
app/reactions/tests.py
View file

@ -101,7 +101,10 @@ class ReactionsViewTest(TestCase):
meta = response.data["meta"]
self.assertEqual(meta["feed"], feed_url)
self.assertEqual(meta["total"], 2)
self.assertIn("version", meta)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_get_reactions_no_reactions(self):
"""Test GET /reactions/ returns empty array for profile with no reactions."""
@ -121,7 +124,10 @@ class ReactionsViewTest(TestCase):
meta = response.data["meta"]
self.assertEqual(meta["feed"], feed_url)
self.assertEqual(meta["total"], 0)
self.assertIn("version", meta)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_get_reactions_nonexistent_profile(self):
"""Test GET /reactions/ returns 404 for nonexistent profile."""

24
app/reactions/utils.py Normal file
View file

@ -0,0 +1,24 @@
"""
Utility functions for reactions
"""
from app.feeds.models import Post
def get_reactions_for_post(post_url: str):
"""
Get all reactions for a specific post.
Args:
post_url: The post URL in format feed_url#post_id
Returns:
QuerySet of Post objects that are reactions to the given post
"""
return (
Post.objects.filter(reply_to=post_url, mood__isnull=False)
.exclude(mood="")
.exclude(poll_votes__isnull=False)
.select_related("profile")
.order_by("-post_id")
)

9
app/reactions/views.py
View file

@ -3,9 +3,9 @@ from rest_framework.response import Response
from rest_framework import status
from django.core.cache import cache
import logging
import hashlib
from app.feeds.models import Profile, Post
from app.reactions.renderers import UTF8JSONRenderer
logger = logging.getLogger(__name__)
@ -13,6 +13,8 @@ logger = logging.getLogger(__name__)
class ReactionsView(APIView):
"""Get reactions for a specific feed URL"""
renderer_classes = [UTF8JSONRenderer]
def get(self, request):
feed_url = request.query_params.get("feed")
@ -77,10 +79,6 @@ class ReactionsView(APIView):
}
reactions_data.append(reaction_data)
# Generate version hash based on profile's last update and reactions count
version_string = f"{profile.last_updated.isoformat()}_{len(reactions_data)}"
version = hashlib.md5(version_string.encode()).hexdigest()[:8]
# URL encode the feed_url for the self link
from urllib.parse import quote
@ -93,7 +91,6 @@ class ReactionsView(APIView):
"meta": {
"feed": feed_url,
"total": len(reactions_data),
"version": version,
},
"_links": {
"self": {

123
app/replies/tests.py
View file

@ -104,7 +104,10 @@ class RepliesViewTest(TestCase):
# Then: Meta should contain correct information
meta = response.data["meta"]
self.assertEqual(meta["parent"], post_url)
self.assertIn("version", meta)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_get_replies_nested_structure(self):
"""Test that replies are properly nested in tree structure."""
@ -191,7 +194,10 @@ class RepliesViewTest(TestCase):
# Then: Meta should still be present
meta = response.data["meta"]
self.assertEqual(meta["parent"], post_url)
self.assertIn("version", meta)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_replies_response_format_compliance(self):
"""Test replies response format compliance with README specification."""
@ -377,3 +383,116 @@ class RepliesViewTest(TestCase):
self.assertIn("moods", reply_tree)
self.assertIsInstance(reply_tree["moods"], list)
self.assertEqual(len(reply_tree["moods"]), 0)
def test_meta_includes_parent_chain(self):
"""Test that meta includes parentChain field for requested post."""
# Given: A post that is a reply (has parents)
# Create a root
root_post = Post.objects.create(
profile=self.profile1,
post_id="2025-01-10T10:00:00+00:00",
content="Root post",
)
# Create a reply to root
reply_post = Post.objects.create(
profile=self.profile2,
post_id="2025-01-10T11:00:00+00:00",
content="Reply to root",
reply_to=f"{self.profile1.feed}#{root_post.post_id}",
)
post_url = f"{self.profile2.feed}#{reply_post.post_id}"
# When: We request replies for the reply_post
response = self.client.get(self.replies_url, {"post": post_url})
# Then: meta should have parentChain
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertIn("meta", response.data)
self.assertIn("parentChain", response.data["meta"])
self.assertIsInstance(response.data["meta"]["parentChain"], list)
# Should have 1 parent (the root)
parent_chain = response.data["meta"]["parentChain"]
self.assertEqual(len(parent_chain), 1)
self.assertEqual(parent_chain[0], f"{self.profile1.feed}#{root_post.post_id}")
def test_parent_chain_empty_for_root_post(self):
"""Test that parentChain is empty array for root posts."""
# Given: A root post (no parents)
post_url = f"{self.profile1.feed}#{self.original_post.post_id}"
# When: We request replies for the root post
response = self.client.get(self.replies_url, {"post": post_url})
# Then: parentChain should be empty
self.assertEqual(response.status_code, status.HTTP_200_OK)
parent_chain = response.data["meta"]["parentChain"]
self.assertEqual(len(parent_chain), 0)
self.assertEqual(parent_chain, [])
def test_parent_chain_order_in_meta(self):
"""Test that parentChain in meta is ordered from root to immediate parent."""
# Given: A deep chain: root -> reply_a -> reply_b -> reply_c
root = Post.objects.create(
profile=self.profile1,
post_id="2025-01-11T10:00:00+00:00",
content="Root",
)
reply_a = Post.objects.create(
profile=self.profile2,
post_id="2025-01-11T11:00:00+00:00",
content="Reply A",
reply_to=f"{self.profile1.feed}#{root.post_id}",
)
reply_b = Post.objects.create(
profile=self.profile3,
post_id="2025-01-11T12:00:00+00:00",
content="Reply B",
reply_to=f"{self.profile2.feed}#{reply_a.post_id}",
)
reply_c = Post.objects.create(
profile=self.profile1,
post_id="2025-01-11T13:00:00+00:00",
content="Reply C",
reply_to=f"{self.profile3.feed}#{reply_b.post_id}",
)
post_url = f"{self.profile1.feed}#{reply_c.post_id}"
# When: We request replies for reply_c
response = self.client.get(self.replies_url, {"post": post_url})
# Then: parentChain should be ordered: root -> reply_a -> reply_b
self.assertEqual(response.status_code, status.HTTP_200_OK)
parent_chain = response.data["meta"]["parentChain"]
self.assertEqual(len(parent_chain), 3)
self.assertEqual(parent_chain[0], f"{self.profile1.feed}#{root.post_id}")
self.assertEqual(parent_chain[1], f"{self.profile2.feed}#{reply_a.post_id}")
self.assertEqual(parent_chain[2], f"{self.profile3.feed}#{reply_b.post_id}")
def test_nodes_do_not_have_parent_chain(self):
"""Test that tree nodes do not include parentChain field."""
# Given: A post with replies
post_url = f"{self.profile1.feed}#{self.original_post.post_id}"
# When: We request replies
response = self.client.get(self.replies_url, {"post": post_url})
# Then: Data nodes should not have parentChain field
self.assertEqual(response.status_code, status.HTTP_200_OK)
data = response.data["data"]
for reply in data:
self.assertNotIn("parentChain", reply)
self.assertNotIn("parent_chain", reply)
# Only post, children, and moods
self.assertIn("post", reply)
self.assertIn("children", reply)
self.assertIn("moods", reply)
# Check nested replies also don't have parentChain
for child in reply["children"]:
self.assertNotIn("parentChain", child)
self.assertNotIn("parent_chain", child)

26
app/replies/utils.py Normal file
View file

@ -0,0 +1,26 @@
"""
Utility functions for replies
"""
from django.db.models import Q
from app.feeds.models import Post
def get_direct_replies_for_post(post_url: str):
"""
Get direct replies for a specific post.
Excludes reactions (posts with mood) and poll votes.
Args:
post_url: The post URL in format feed_url#post_id
Returns:
QuerySet of Post objects that are direct replies to the given post
"""
return (
Post.objects.filter(reply_to=post_url)
.filter(Q(mood="") | Q(mood__isnull=True))
.exclude(poll_votes__isnull=False)
.select_related("profile")
.order_by("-post_id")
)

15
app/replies/views.py
View file

@ -3,9 +3,9 @@ from rest_framework.response import Response
from rest_framework import status
from django.core.cache import cache
import logging
import hashlib
from app.feeds.models import Post, Profile
from app.feeds.utils import get_parent_chain
logger = logging.getLogger(__name__)
@ -97,9 +97,8 @@ class RepliesView(APIView):
# Find moods for the original post
moods = self._find_moods(original_post_url, replies)
# Generate version hash
version_string = f"{original_post.updated_at.isoformat()}_{len(replies)}"
version = hashlib.md5(version_string.encode()).hexdigest()[:8]
# Calculate parent chain for the original post
parent_chain = get_parent_chain(original_post)
# URL encode the post_url for the self link
from urllib.parse import quote
@ -112,8 +111,8 @@ class RepliesView(APIView):
"data": replies_tree,
"meta": {
"parent": original_post_url,
"version": version,
"moods": moods,
"parentChain": parent_chain,
},
"_links": {
"self": {"href": f"/replies/?post={encoded_post_url}", "method": "GET"}
@ -151,7 +150,11 @@ class RepliesView(APIView):
# Find moods for this reply
moods = self._find_moods(reply_url, all_replies)
reply_node = {"post": reply_url, "children": children, "moods": moods}
reply_node = {
"post": reply_url,
"children": children,
"moods": moods,
}
result.append(reply_node)
return result

10
app/repliesto/tests.py
View file

@ -107,7 +107,10 @@ class RepliesToViewTest(TestCase):
meta = response.data["meta"]
self.assertEqual(meta["feed"], feed_url)
self.assertEqual(meta["total"], 2)
self.assertIn("version", meta)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_get_replies_no_replies(self):
"""Test GET /replies-to/ returns empty array for profile with no replies."""
@ -127,7 +130,10 @@ class RepliesToViewTest(TestCase):
meta = response.data["meta"]
self.assertEqual(meta["feed"], feed_url)
self.assertEqual(meta["total"], 0)
self.assertIn("version", meta)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_get_replies_nonexistent_profile(self):
"""Test GET /replies-to/ returns 404 for nonexistent profile."""

7
app/repliesto/views.py
View file

@ -4,7 +4,6 @@ from rest_framework import status
from django.core.cache import cache
from django.db.models import Q
import logging
import hashlib
from app.feeds.models import Profile, Post
@ -78,10 +77,6 @@ class RepliesToView(APIView):
}
replies_data.append(reply_data)
# Generate version hash based on profile's last update and replies count
version_string = f"{profile.last_updated.isoformat()}_{len(replies_data)}"
version = hashlib.md5(version_string.encode()).hexdigest()[:8]
# URL encode the feed_url for the self link
from urllib.parse import quote
@ -91,7 +86,7 @@ class RepliesToView(APIView):
"type": "Success",
"errors": [],
"data": replies_data,
"meta": {"feed": feed_url, "total": len(replies_data), "version": version},
"meta": {"feed": feed_url, "total": len(replies_data)},
"_links": {
"self": {
"href": f"/replies-to/?feed={encoded_feed_url}",

0
app/rss/__init__.py Normal file
View file

6
app/rss/apps.py Normal file
View file

@ -0,0 +1,6 @@
from django.apps import AppConfig
class RssConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "app.rss"

541
app/rss/tests.py Normal file
View file

@ -0,0 +1,541 @@
from django.test import TestCase, override_settings
from rest_framework.test import APIClient
from rest_framework import status
from xml.etree import ElementTree as ET
from app.feeds.models import Profile, Post
class RSSFeedTest(TestCase):
"""Test cases for the RSS Feed using Given/When/Then structure."""
def setUp(self):
self.client = APIClient()
self.rss_url = "/rss.xml"
# Create test profiles
self.profile1 = Profile.objects.create(
feed="https://example.com/social.org",
title="Example Profile",
nick="example_user",
description="Test profile 1",
)
self.profile2 = Profile.objects.create(
feed="https://test.com/social.org",
title="Test Profile",
nick="test_user",
description="Test profile 2",
)
self.profile3 = Profile.objects.create(
feed="https://third.com/social.org",
title="Third Profile",
nick="third_user",
description="Test profile 3",
)
# Create test posts with different content and tags
self.post1 = Post.objects.create(
profile=self.profile1,
post_id="2025-01-01T12:00:00+00:00",
content="This post is about Emacs and org-mode",
tags="emacs org-mode",
)
self.post2 = Post.objects.create(
profile=self.profile2,
post_id="2025-01-01T13:00:00+00:00",
content="Learning Python programming language",
tags="python programming",
)
self.post3 = Post.objects.create(
profile=self.profile3,
post_id="2025-01-01T14:00:00+00:00",
content="Django web framework with Python",
tags="django python web",
)
self.post4 = Post.objects.create(
profile=self.profile1,
post_id="2025-01-01T15:00:00+00:00",
content="Emacs configuration and setup",
tags="emacs configuration",
)
self.post5 = Post.objects.create(
profile=self.profile2,
post_id="2025-01-01T16:00:00+00:00",
content="JavaScript and React development",
tags="javascript react frontend",
)
def test_rss_feed_all_posts_success(self):
"""Test GET /rss.xml returns RSS feed with all posts."""
# Given: Posts exist in the database
# When: We request the RSS feed
response = self.client.get(self.rss_url)
# Then: Should return 200 and valid RSS
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response["Content-Type"], "application/rss+xml; charset=utf-8")
# Parse XML and verify structure
root = ET.fromstring(response.content)
self.assertEqual(root.tag, "rss")
self.assertEqual(root.get("version"), "2.0")
# Verify channel
channel = root.find("channel")
self.assertIsNotNone(channel)
# Verify title
title = channel.find("title")
self.assertIsNotNone(title)
self.assertEqual(title.text, "Org Social Relay - Latest Posts")
# Verify items
items = channel.findall("item")
self.assertEqual(len(items), 5) # All 5 posts
# Verify first item (most recent)
first_item = items[0]
guid = first_item.find("guid")
self.assertIsNotNone(guid)
self.assertIn(self.post5.post_id, guid.text)
def test_rss_feed_filtered_by_tag(self):
"""Test GET /rss.xml?tag=<tag> returns RSS feed filtered by tag."""
# Given: Posts with various tags
# When: We request the RSS feed filtered by tag "emacs"
response = self.client.get(self.rss_url, {"tag": "emacs"})
# Then: Should return 200 and filtered RSS
self.assertEqual(response.status_code, status.HTTP_200_OK)
# Parse XML
root = ET.fromstring(response.content)
channel = root.find("channel")
# Verify title includes tag
title = channel.find("title")
self.assertIn("emacs", title.text.lower())
# Verify only posts with "emacs" tag are included
items = channel.findall("item")
self.assertEqual(len(items), 2) # post1 and post4
# Verify all items have the emacs tag
for item in items:
categories = [cat.text for cat in item.findall("category")]
self.assertIn("emacs", categories)
def test_rss_feed_filtered_by_feed(self):
"""Test GET /rss.xml?feed=<feed_url> returns RSS feed filtered by author feed."""
# Given: Posts from different profiles
# When: We request the RSS feed filtered by profile1's feed
response = self.client.get(self.rss_url, {"feed": self.profile1.feed})
# Then: Should return 200 and filtered RSS
self.assertEqual(response.status_code, status.HTTP_200_OK)
# Parse XML
root = ET.fromstring(response.content)
channel = root.find("channel")
# Verify title includes feed
title = channel.find("title")
self.assertIn(self.profile1.feed, title.text)
# Verify only posts from profile1 are included
items = channel.findall("item")
self.assertEqual(len(items), 2) # post1 and post4
# Verify all items are from profile1
for item in items:
link = item.find("link")
self.assertIn(self.profile1.feed, link.text)
def test_rss_feed_item_structure(self):
"""Test that RSS feed items have correct structure."""
# Given: Posts exist
# When: We request the RSS feed
response = self.client.get(self.rss_url)
# Parse XML
root = ET.fromstring(response.content)
channel = root.find("channel")
items = channel.findall("item")
# Then: Each item should have required fields
for item in items:
# Required fields
self.assertIsNotNone(item.find("title"))
self.assertIsNotNone(item.find("link"))
self.assertIsNotNone(item.find("description"))
self.assertIsNotNone(item.find("pubDate"))
self.assertIsNotNone(item.find("guid"))
self.assertIsNotNone(item.find("author"))
def test_rss_feed_limit_200_posts(self):
"""Test that RSS feed is limited to 200 posts as per specification."""
# Given: More than 200 posts
for i in range(201):
Post.objects.create(
profile=self.profile1,
post_id=f"2025-01-02T{i // 60:02d}:{i % 60:02d}:00+00:00",
content=f"Post number {i}",
tags="test",
)
# When: We request the RSS feed
response = self.client.get(self.rss_url)
# Parse XML
root = ET.fromstring(response.content)
channel = root.find("channel")
items = channel.findall("item")
# Then: Should be limited to 200 posts
self.assertEqual(len(items), 200)
def test_rss_feed_ordered_by_most_recent(self):
"""Test that RSS feed items are ordered from most recent to oldest."""
# Given: Posts with different timestamps
# When: We request the RSS feed
response = self.client.get(self.rss_url)
# Parse XML
root = ET.fromstring(response.content)
channel = root.find("channel")
items = channel.findall("item")
# Then: First item should be the most recent post
first_item = items[0]
first_guid = first_item.find("guid").text
self.assertIn(self.post5.post_id, first_guid) # Most recent
# Last item should be the oldest post
last_item = items[-1]
last_guid = last_item.find("guid").text
self.assertIn(self.post1.post_id, last_guid) # Oldest
def test_rss_feed_categories_from_tags(self):
"""Test that post tags are converted to RSS categories."""
# Given: Post with multiple tags
# When: We request the RSS feed
response = self.client.get(self.rss_url)
# Parse XML
root = ET.fromstring(response.content)
channel = root.find("channel")
items = channel.findall("item")
# Find item for post3 (has multiple tags)
post3_item = None
for item in items:
guid = item.find("guid").text
if self.post3.post_id in guid:
post3_item = item
break
self.assertIsNotNone(post3_item)
# Then: Categories should match tags
categories = [cat.text for cat in post3_item.findall("category")]
expected_tags = self.post3.tags.split()
self.assertEqual(set(categories), set(expected_tags))
def test_rss_feed_guid_is_post_url(self):
"""Test that GUID is the post URL."""
# Given: Posts exist
# When: We request the RSS feed
response = self.client.get(self.rss_url)
# Parse XML
root = ET.fromstring(response.content)
channel = root.find("channel")
items = channel.findall("item")
# Then: Each GUID should be feed#post_id format
for item in items:
guid = item.find("guid")
self.assertIsNotNone(guid)
self.assertIn("#", guid.text)
self.assertTrue(guid.get("isPermaLink") in ["true", None])
def test_rss_feed_content_type(self):
"""Test that RSS feed returns correct content type."""
# Given: RSS endpoint
# When: We request the RSS feed
response = self.client.get(self.rss_url)
# Then: Content-Type should be application/rss+xml
self.assertEqual(response["Content-Type"], "application/rss+xml; charset=utf-8")
def test_rss_feed_valid_xml(self):
"""Test that RSS feed returns valid XML."""
# Given: RSS endpoint
# When: We request the RSS feed
response = self.client.get(self.rss_url)
# Then: Should be parseable as XML without errors
try:
ET.fromstring(response.content)
except ET.ParseError as e:
self.fail(f"RSS feed is not valid XML: {e}")
def test_rss_feed_description_is_post_content(self):
"""Test that description contains the post content (converted to HTML)."""
# Given: Posts with content
# When: We request the RSS feed
response = self.client.get(self.rss_url)
# Parse XML
root = ET.fromstring(response.content)
channel = root.find("channel")
items = channel.findall("item")
# Find item for post1
post1_item = None
for item in items:
guid = item.find("guid").text
if self.post1.post_id in guid:
post1_item = item
break
self.assertIsNotNone(post1_item)
# Then: Description should contain HTML-converted content
description = post1_item.find("description")
self.assertIsNotNone(description.text)
# Content is now converted to HTML, so we check for HTML elements
self.assertIn("<p>", description.text)
# The original content should be present in the HTML
self.assertIn("This post is about Emacs and org-mode", description.text)
def test_rss_feed_author_is_nick(self):
"""Test that author is the profile nick."""
# Given: Posts from different profiles
# When: We request the RSS feed
response = self.client.get(self.rss_url)
# Parse XML
root = ET.fromstring(response.content)
channel = root.find("channel")
items = channel.findall("item")
# Find item for post1
post1_item = None
for item in items:
guid = item.find("guid").text
if self.post1.post_id in guid:
post1_item = item
break
self.assertIsNotNone(post1_item)
# Then: Author should be the profile nick
author = post1_item.find("author")
self.assertIn(self.profile1.nick, author.text)
def test_rss_feed_empty_when_no_posts(self):
"""Test RSS feed when there are no posts."""
# Given: No posts exist
Post.objects.all().delete()
# When: We request the RSS feed
response = self.client.get(self.rss_url)
# Then: Should still return valid RSS with no items
self.assertEqual(response.status_code, status.HTTP_200_OK)
# Parse XML
root = ET.fromstring(response.content)
channel = root.find("channel")
items = channel.findall("item")
# Should have 0 items
self.assertEqual(len(items), 0)
def test_rss_feed_link_is_post_url(self):
"""Test that link element is the post URL."""
# Given: Posts exist
# When: We request the RSS feed
response = self.client.get(self.rss_url)
# Parse XML
root = ET.fromstring(response.content)
channel = root.find("channel")
items = channel.findall("item")
# Then: Each link should be feed#post_id format
for item in items:
link = item.find("link")
guid = item.find("guid")
# Link and GUID should be the same
self.assertEqual(link.text, guid.text)
self.assertIn("#", link.text)
def test_rss_feed_excludes_empty_content(self):
"""Test that RSS feed excludes posts with empty content (reactions, votes, etc.)."""
# Given: Posts with and without content
# Create posts with empty content (reactions, votes)
Post.objects.create(
profile=self.profile1,
post_id="2025-01-01T17:00:00+00:00",
content="", # Empty content (reaction/vote)
tags="reaction",
)
Post.objects.create(
profile=self.profile2,
post_id="2025-01-01T18:00:00+00:00",
content=" ", # Whitespace only
tags="vote",
)
# When: We request the RSS feed
response = self.client.get(self.rss_url)
# Parse XML
root = ET.fromstring(response.content)
channel = root.find("channel")
items = channel.findall("item")
# Then: Should only include posts with content (5 original posts)
# Empty and whitespace-only posts should be excluded
self.assertEqual(len(items), 5)
# Verify all items have non-empty description
for item in items:
description = item.find("description")
# Description should exist and have content
self.assertIsNotNone(description.text)
# After HTML conversion, should have actual content
self.assertIn("<p>", description.text)
def test_rss_feed_excludes_bridge_posts(self):
"""Test that posts from bridge virtual feeds are excluded from the feed."""
# Given: A bridge profile with a post, alongside the regular posts
bridge_profile = Profile.objects.create(
feed=(
"https://relay.org-social.org/bridge/rss/"
"?url=https%3A%2F%2Fexample.org%2Ffeed.xml"
),
title="Bridged Feed",
nick="bridged_feed",
description="Bridge virtual feed",
)
bridge_post = Post.objects.create(
profile=bridge_profile,
post_id="2025-01-01T20:00:00+00:00",
content="Post coming from a bridged RSS feed",
tags="bridged",
)
# When: We request the global RSS feed
response = self.client.get(self.rss_url)
# Then: Only the posts from real profiles are included
root = ET.fromstring(response.content)
items = root.find("channel").findall("item")
self.assertEqual(len(items), 5)
guids = [item.find("guid").text for item in items]
for guid in guids:
self.assertNotIn(bridge_post.post_id, guid)
def test_rss_feed_filtered_by_tag_excludes_bridge_posts(self):
"""Test that bridge posts are excluded from tag-filtered feeds too."""
# Given: A bridge profile with a post tagged like a regular post
bridge_profile = Profile.objects.create(
feed="https://relay.org-social.org/bridge/activitypub/@user@instance.tld/",
title="Bridged Account",
nick="bridged_account",
description="Bridge virtual feed",
)
Post.objects.create(
profile=bridge_profile,
post_id="2025-01-01T21:00:00+00:00",
content="Bridged post about Emacs",
tags="emacs",
)
# When: We request the RSS feed filtered by tag "emacs"
response = self.client.get(self.rss_url, {"tag": "emacs"})
# Then: Only the real posts with that tag are included
root = ET.fromstring(response.content)
items = root.find("channel").findall("item")
self.assertEqual(len(items), 2) # post1 and post4
def test_rss_feed_explicit_bridge_feed_is_still_served(self):
"""Test that explicitly requesting a bridge feed still returns its posts."""
# Given: A bridge profile with a post
bridge_feed_url = (
"https://relay.org-social.org/bridge/rss/"
"?url=https%3A%2F%2Fexample.org%2Ffeed.xml"
)
bridge_profile = Profile.objects.create(
feed=bridge_feed_url,
title="Bridged Feed",
nick="bridged_feed",
description="Bridge virtual feed",
)
bridge_post = Post.objects.create(
profile=bridge_profile,
post_id="2025-01-01T20:00:00+00:00",
content="Post coming from a bridged RSS feed",
tags="bridged",
)
# When: We request the RSS feed filtered by the bridge feed itself
response = self.client.get(self.rss_url, {"feed": bridge_feed_url})
# Then: The bridge post is served
root = ET.fromstring(response.content)
items = root.find("channel").findall("item")
self.assertEqual(len(items), 1)
self.assertIn(bridge_post.post_id, items[0].find("guid").text)
@override_settings(
CACHES={
"default": {
"BACKEND": "django.core.cache.backends.locmem.LocMemCache",
"LOCATION": "test-cache",
}
}
)
def test_rss_feed_xml_caching(self):
"""Test that RSS feed XML response is cached for performance."""
from django.core.cache import cache
# Given: Posts exist and cache is cleared
cache_key = "rss_xml_all"
cache.delete(cache_key)
# When: We request the RSS feed for the first time
response1 = self.client.get(self.rss_url)
self.assertEqual(response1.status_code, status.HTTP_200_OK)
# Then: The XML should be cached
cached_xml = cache.get(cache_key)
self.assertIsNotNone(cached_xml)
# When: We request the RSS feed again
response2 = self.client.get(self.rss_url)
self.assertEqual(response2.status_code, status.HTTP_200_OK)
# Then: Both responses should be identical
self.assertEqual(response1.content, response2.content)
self.assertEqual(response1["Content-Type"], response2["Content-Type"])

6
app/rss/urls.py Normal file
View file

@ -0,0 +1,6 @@
from django.urls import path
from .views import LatestPostsFeed
urlpatterns = [
path("", LatestPostsFeed(), name="rss_feed"),
]

238
app/rss/views.py Normal file
View file

@ -0,0 +1,238 @@
from django.contrib.syndication.views import Feed
from django.utils.feedgenerator import Rss201rev2Feed
from django.core.cache import cache
from django.conf import settings
from django.http import HttpResponse
import hashlib
import re
import logging
from app.bridge.models import bridge_urls_q
from app.feeds.models import Post
try:
from orgpython import to_html as org_to_html
HAS_ORGPYTHON = True
except ImportError:
HAS_ORGPYTHON = False
logger = logging.getLogger(__name__)
class CustomRss201rev2Feed(Rss201rev2Feed):
"""Custom RSS generator that adds author field properly"""
def add_item_elements(self, handler, item):
"""Add item elements including author"""
super().add_item_elements(handler, item)
# Add author field if provided
if item.get("author"):
handler.addQuickElement("author", item["author"])
class LatestPostsFeed(Feed):
"""RSS feed for latest posts from Org Social Relay"""
feed_type = CustomRss201rev2Feed
def __call__(self, request, *args, **kwargs):
"""Override to add XML caching"""
# Build cache key for the XML response
tag = request.GET.get("tag", "")
feed_url = request.GET.get("feed", "")
cache_parts = ["rss_xml"]
if tag:
cache_parts.append(f"tag_{hashlib.md5(tag.encode()).hexdigest()[:8]}")
elif feed_url:
cache_parts.append(f"feed_{hashlib.md5(feed_url.encode()).hexdigest()[:8]}")
else:
cache_parts.append("all")
cache_key = "_".join(cache_parts)
# Try to get cached XML
cached_xml = cache.get(cache_key)
if cached_xml is not None:
return HttpResponse(
cached_xml, content_type="application/rss+xml; charset=utf-8"
)
# Generate feed normally
response = super().__call__(request, *args, **kwargs)
# Cache the XML response for 5 minutes (300 seconds)
if response.status_code == 200:
cache.set(cache_key, response.content, 300)
return response
def get_object(self, request):
"""Process query parameters"""
tag = request.GET.get("tag")
feed_url = request.GET.get("feed")
return {"tag": tag, "feed": feed_url}
def title(self, obj):
"""Generate feed title based on filters"""
if obj["tag"]:
return f"Org Social Relay - Posts tagged with '{obj['tag']}'"
elif obj["feed"]:
return f"Org Social Relay - Posts from {obj['feed']}"
return "Org Social Relay - Latest Posts"
def link(self, obj):
"""Generate feed link"""
site_domain = settings.SITE_DOMAIN
protocol = "https" if not settings.DEBUG else "http"
base_url = f"{protocol}://{site_domain}/rss.xml"
if obj["tag"]:
return f"{base_url}?tag={obj['tag']}"
elif obj["feed"]:
return f"{base_url}?feed={obj['feed']}"
return base_url
def description(self, obj):
"""Generate feed description based on filters"""
if obj["tag"]:
return f"Latest posts from Org Social Relay tagged with '{obj['tag']}'"
elif obj["feed"]:
return f"Latest posts from {obj['feed']} on Org Social Relay"
return "Latest posts from all registered feeds on Org Social Relay"
def items(self, obj):
"""Return items for the feed, limited to 200 posts"""
# Build cache key based on filters
cache_parts = ["rss_feed"]
if obj["tag"]:
cache_parts.append(
f"tag_{hashlib.md5(obj['tag'].encode()).hexdigest()[:8]}"
)
elif obj["feed"]:
cache_parts.append(
f"feed_{hashlib.md5(obj['feed'].encode()).hexdigest()[:8]}"
)
else:
cache_parts.append("all")
cache_key = "_".join(cache_parts)
cached_posts = cache.get(cache_key)
if cached_posts is not None:
return cached_posts
# Build query - exclude posts with empty content (reactions, votes, etc.)
posts_query = (
Post.objects.select_related("profile")
.exclude(content__isnull=True)
.exclude(content__exact="")
.order_by("-created_at")
)
# Apply filters
if obj["tag"]:
# Search by specific tag (exact word match, case insensitive)
tag_escaped = re.escape(obj["tag"])
tag_pattern = rf"(^|[\s]){tag_escaped}([\s]|$)"
posts_query = posts_query.filter(tags__iregex=tag_pattern).exclude(
bridge_urls_q("profile__feed")
)
elif obj["feed"]:
# Filter by author feed. An explicitly requested bridge feed is
# still served: only aggregate feeds must leave bridges out.
posts_query = posts_query.filter(profile__feed=obj["feed"])
else:
posts_query = posts_query.exclude(bridge_urls_q("profile__feed"))
# Fetch more than 200 to account for filtering whitespace-only posts
posts_raw = list(posts_query[:250])
# Filter out posts with only whitespace content
posts = [p for p in posts_raw if p.content and p.content.strip()]
# Limit to 200 posts as per specification
posts = posts[:200]
# Cache permanently (will be cleared by scan_feeds task)
cache.set(cache_key, posts, None)
return posts
def item_title(self, item):
"""Generate item title: date + nick"""
# Format: "2025-11-15 - username"
date_str = item.created_at.strftime("%Y-%m-%d")
return f"{date_str} - {item.profile.nick}"
def item_description(self, item):
"""Return full post content as description, converted from Org to HTML"""
if not item.content:
return ""
# Convert Org mode content to HTML if org-python is available
if HAS_ORGPYTHON:
try:
# Posts in Org Social start at level 3 (***), but in RSS context
# they should start at level 1. Remove 2 asterisks from headings
# before conversion: *** -> *, **** -> **, etc.
content = re.sub(
r"^(\*{2,})",
lambda m: "*" * (len(m.group(1)) - 2),
item.content,
flags=re.MULTILINE,
)
# Convert org-mode to HTML
# We disable toc as we're showing individual posts
# highlight=True enables syntax highlighting for code blocks
html_content = org_to_html(content, toc=False, highlight=True)
return html_content
except Exception as e:
logger.warning(
f"Failed to convert Org to HTML for post {item.post_id}: {e}"
)
# Fallback to plain text
return item.content
# If org-python is not available, return plain text
return item.content
def item_link(self, item):
"""Generate item link (post URL)"""
return f"{item.profile.feed}#{item.post_id}"
def item_author_name(self, item):
"""Return author name"""
return item.profile.nick
def item_author_email(self, item):
"""Return author email - using feed URL as identifier"""
# RSS 2.0 requires email format for author, but we don't have emails
# We'll return None and handle it differently
return None
def item_extra_kwargs(self, item):
"""Add custom author field"""
return {"author": item.profile.nick}
def item_pubdate(self, item):
"""Return publication date"""
return item.created_at
def item_categories(self, item):
"""Return post tags as categories"""
if item.tags:
return item.tags.split()
return []
def item_guid(self, item):
"""Return unique identifier for the item"""
return f"{item.profile.feed}#{item.post_id}"
def item_guid_is_permalink(self, item):
"""The GUID is a permalink"""
return True

10
app/search/tests.py
View file

@ -112,7 +112,10 @@ class SearchViewTest(TestCase):
self.assertEqual(meta["perPage"], 10)
self.assertFalse(meta["hasNext"])
self.assertFalse(meta["hasPrevious"])
self.assertIn("version", meta)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
def test_search_by_tag_success(self):
"""Test GET /search/?tag=<tag> returns posts with specific tag."""
@ -278,7 +281,6 @@ class SearchViewTest(TestCase):
# Then: Meta should contain all required fields
meta = response.data["meta"]
required_meta_fields = [
"version",
"query",
"total",
"page",
@ -289,6 +291,10 @@ class SearchViewTest(TestCase):
for field in required_meta_fields:
self.assertIn(field, meta)
# Then: Should have ETag and Last-Modified headers
self.assertIn("ETag", response)
self.assertIn("Last-Modified", response)
# Check that _links exists
self.assertIn("_links", response.data)
links = response.data["_links"]

5
app/search/views.py
View file

@ -108,17 +108,12 @@ class SearchView(APIView):
if per_page != 10:
base_url += f"&perPage={per_page}"
# Generate version hash based on search parameters and total results
version_string = f"{search_term}_{total_posts}_{posts_query.first().updated_at.isoformat() if posts_query.exists() else 'empty'}"
version = hashlib.md5(version_string.encode()).hexdigest()[:8]
# Build response
response_data = {
"type": "Success",
"errors": [],
"data": data,
"meta": {
"version": version,
search_type: search_value,
"total": total_posts,
"page": page,

1
app/sse_notifications/__init__.py Normal file
View file

@ -0,0 +1 @@
# SSE Notifications app

6
app/sse_notifications/apps.py Normal file
View file

@ -0,0 +1,6 @@
from django.apps import AppConfig
class SseNotificationsConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "app.sse_notifications"

359
app/sse_notifications/tests.py Normal file
View file

@ -0,0 +1,359 @@
import asyncio
import json
import pytest
from unittest.mock import patch, AsyncMock, MagicMock
from django.test import TestCase, Client
from app.feeds.notification_publisher import publish_notification
def collect_stream(response):
"""Consume an async streaming response and return the decoded content."""
async def _collect():
chunks = []
async for chunk in response.streaming_content:
if isinstance(chunk, bytes):
chunks.append(chunk)
else:
chunks.append(chunk.encode("utf-8"))
return b"".join(chunks).decode("utf-8")
return asyncio.run(_collect())
def make_async_pubsub(messages=None, error=None):
"""Build an async pubsub mock that returns messages then stops."""
mock_pubsub = AsyncMock()
if error:
mock_pubsub.get_message.side_effect = error
elif messages is not None:
# Yield each message, then raise RedisError to break the loop
import redis.asyncio as aioredis
mock_pubsub.get_message.side_effect = messages + [aioredis.RedisError("done")]
else:
import redis.asyncio as aioredis
mock_pubsub.get_message.side_effect = aioredis.RedisError("done")
return mock_pubsub
class TestSSENotificationsEndpoint(TestCase):
def setUp(self):
"""Set up test fixtures."""
self.client = Client()
self.feed_url = "https://example.com/social.org"
def _patch_redis(self, pubsub):
mock_redis = AsyncMock()
mock_redis.pubsub = MagicMock(return_value=pubsub) # pubsub() is a sync call
return patch("app.sse_notifications.views.aioredis.Redis", return_value=mock_redis)
def test_sse_endpoint_without_feed_returns_global_stream(self):
"""Test that SSE endpoint without feed parameter returns global stream."""
# Given: A Redis pubsub with no pending messages
pubsub = make_async_pubsub()
with self._patch_redis(pubsub):
# When: The SSE endpoint is requested without a feed parameter
response = self.client.get("/sse/notifications/")
# Then: An event stream response is returned
self.assertEqual(response.status_code, 200)
self.assertEqual(response["Content-Type"], "text/event-stream")
def test_sse_endpoint_accepts_valid_feed(self):
"""Test that SSE endpoint accepts valid feed parameter."""
# Given: A Redis pubsub with no pending messages
pubsub = make_async_pubsub()
with self._patch_redis(pubsub):
# When: The SSE endpoint is requested for a specific feed
response = self.client.get("/sse/notifications/", {"feed": self.feed_url})
# Then: An event stream response is returned with anti-buffering headers
self.assertEqual(response.status_code, 200)
self.assertEqual(response["Content-Type"], "text/event-stream")
self.assertEqual(response["Cache-Control"], "no-cache")
self.assertEqual(response["X-Accel-Buffering"], "no")
self.assertEqual(response["Access-Control-Allow-Origin"], "*")
def test_sse_sends_connection_event_with_feed(self):
"""Test that per-feed SSE sends initial connection event with feed field."""
# Given: A Redis pubsub with no pending messages
pubsub = make_async_pubsub()
with self._patch_redis(pubsub):
# When: The per-feed stream is requested and consumed
response = self.client.get("/sse/notifications/", {"feed": self.feed_url})
content = collect_stream(response)
# Then: The stream starts with a connection event naming the feed
self.assertIn("event: connected", content)
self.assertIn(f'"feed": "{self.feed_url}"', content)
self.assertIn('"status": "connected"', content)
def test_sse_sends_connection_event_global(self):
"""Test that global SSE sends initial connection event without feed field."""
# Given: A Redis pubsub with no pending messages
pubsub = make_async_pubsub()
with self._patch_redis(pubsub):
# When: The global stream is requested and consumed
response = self.client.get("/sse/notifications/")
content = collect_stream(response)
# Then: The stream starts with a connection event without a feed field
self.assertIn("event: connected", content)
self.assertIn('"status": "connected"', content)
self.assertNotIn('"feed":', content)
def test_sse_receives_notification_from_redis(self):
"""Test that per-feed SSE receives and forwards notifications from Redis."""
# Given: A Redis pubsub holding one mention notification
notification_data = {
"type": "mention",
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
}
pubsub = make_async_pubsub(
messages=[{"type": "message", "data": json.dumps(notification_data)}]
)
with self._patch_redis(pubsub):
# When: The per-feed stream is requested and consumed
response = self.client.get("/sse/notifications/", {"feed": self.feed_url})
content = collect_stream(response)
# Then: The notification is forwarded as an SSE event
self.assertIn("event: notification", content)
self.assertIn('"type": "mention"', content)
self.assertIn("https://alice.com/social.org#2024-01-01T10:00:00+0000", content)
def test_global_sse_adds_target_feed_to_notifications(self):
"""Test that global SSE adds target_feed field extracted from channel name."""
# Given: A Redis pubsub holding one pattern message for a feed channel
notification_data = {
"type": "mention",
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
}
target_feed = "https://example.com/social.org"
pubsub = make_async_pubsub(
messages=[
{
"type": "pmessage",
"pattern": "notifications:*",
"channel": f"notifications:{target_feed}",
"data": json.dumps(notification_data),
}
]
)
with self._patch_redis(pubsub):
# When: The global stream is requested and consumed
response = self.client.get("/sse/notifications/")
content = collect_stream(response)
# Then: The forwarded event includes the target_feed from the channel
self.assertIn("event: notification", content)
self.assertIn(f'"target_feed": "{target_feed}"', content)
self.assertIn('"type": "mention"', content)
def test_global_sse_uses_psubscribe(self):
"""Test that global SSE subscribes with psubscribe to all notification channels."""
# Given: A Redis pubsub with no pending messages
pubsub = make_async_pubsub()
with self._patch_redis(pubsub):
# When: The global stream is requested and consumed
response = self.client.get("/sse/notifications/")
collect_stream(response)
# Then: The view subscribed by pattern to every notification channel
pubsub.psubscribe.assert_awaited_once_with("notifications:*")
def test_per_feed_sse_uses_subscribe(self):
"""Test that per-feed SSE subscribes with subscribe to the specific channel."""
# Given: A Redis pubsub with no pending messages
pubsub = make_async_pubsub()
with self._patch_redis(pubsub):
# When: The per-feed stream is requested and consumed
response = self.client.get("/sse/notifications/", {"feed": self.feed_url})
collect_stream(response)
# Then: The view subscribed only to that feed's channel
pubsub.subscribe.assert_awaited_once_with(f"notifications:{self.feed_url}")
class TestNotificationPublisher(TestCase):
"""Test the notification publisher module"""
@patch("app.feeds.notification_publisher.redis.Redis")
def test_publish_mention_notification(self, mock_redis):
"""Test publishing a mention notification"""
# Given: A working Redis connection and a mention to notify
mock_redis_instance = MagicMock()
mock_redis.return_value = mock_redis_instance
target_feed = "https://bob.com/social.org"
post_url = "https://alice.com/social.org#2024-01-01T10:00:00+0000"
# When: The mention notification is published
result = publish_notification(
target_feed_url=target_feed, notification_type="mention", post_url=post_url
)
# Then: One message is published on the target feed's channel
self.assertTrue(result)
mock_redis_instance.publish.assert_called_once()
call_args = mock_redis_instance.publish.call_args
channel, data = call_args[0]
self.assertEqual(channel, f"notifications:{target_feed}")
notification = json.loads(data)
self.assertEqual(notification["type"], "mention")
self.assertEqual(notification["post"], post_url)
@patch("app.feeds.notification_publisher.redis.Redis")
def test_publish_reaction_notification_with_emoji(self, mock_redis):
"""Test publishing a reaction notification with emoji"""
# Given: A working Redis connection and a reaction to notify
mock_redis_instance = MagicMock()
mock_redis.return_value = mock_redis_instance
# When: The reaction notification is published
result = publish_notification(
target_feed_url="https://bob.com/social.org",
notification_type="reaction",
post_url="https://alice.com/social.org#2024-01-01T10:00:00+0000",
emoji="❤",
parent="https://bob.com/social.org#2024-01-01T09:00:00+0000",
)
# Then: The published message carries the reaction type and emoji
self.assertTrue(result)
call_args = mock_redis_instance.publish.call_args
notification = json.loads(call_args[0][1])
self.assertEqual(notification["type"], "reaction")
self.assertEqual(notification["emoji"], "❤")
@patch("app.feeds.notification_publisher.redis.Redis")
def test_publish_reply_notification(self, mock_redis):
"""Test publishing a reply notification"""
# Given: A working Redis connection and a reply to notify
mock_redis_instance = MagicMock()
mock_redis.return_value = mock_redis_instance
# When: The reply notification is published
result = publish_notification(
target_feed_url="https://bob.com/social.org",
notification_type="reply",
post_url="https://alice.com/social.org#2024-01-01T10:00:00+0000",
parent="https://bob.com/social.org#2024-01-01T09:00:00+0000",
)
# Then: The published message carries the reply type
self.assertTrue(result)
notification = json.loads(mock_redis_instance.publish.call_args[0][1])
self.assertEqual(notification["type"], "reply")
@patch("app.feeds.notification_publisher.redis.Redis")
def test_publish_boost_notification(self, mock_redis):
"""Test publishing a boost notification"""
# Given: A working Redis connection and a boost to notify
mock_redis_instance = MagicMock()
mock_redis.return_value = mock_redis_instance
# When: The boost notification is published
result = publish_notification(
target_feed_url="https://bob.com/social.org",
notification_type="boost",
post_url="https://alice.com/social.org#2024-01-01T10:00:00+0000",
boosted="https://bob.com/social.org#2024-01-01T09:00:00+0000",
)
# Then: The published message carries the boost type
self.assertTrue(result)
notification = json.loads(mock_redis_instance.publish.call_args[0][1])
self.assertEqual(notification["type"], "boost")
@patch("app.feeds.notification_publisher.redis.Redis")
def test_publish_notification_handles_redis_error(self, mock_redis):
"""Test that publish_notification handles Redis errors gracefully"""
# Given: A Redis connection that fails
mock_redis.side_effect = Exception("Redis connection failed")
# When: A notification is published
result = publish_notification(
target_feed_url="https://bob.com/social.org",
notification_type="mention",
post_url="https://alice.com/social.org#2024-01-01T10:00:00+0000",
)
# Then: The failure is reported without raising
self.assertFalse(result)
@pytest.mark.django_db
class TestSSENotificationStructure:
"""Test that SSE notifications match the expected JSON structure"""
def test_mention_notification_structure(self):
# Given: A mention notification payload
notification = {
"type": "mention",
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
}
# Then: It carries the mention type and a post URL with fragment
assert "type" in notification
assert notification["type"] == "mention"
assert "#" in notification["post"]
def test_reaction_notification_structure(self):
# Given: A reaction notification payload
notification = {
"type": "reaction",
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
"emoji": "❤",
"parent": "https://bob.com/social.org#2024-01-01T09:00:00+0000",
}
# Then: It carries the reaction type, emoji and parent
assert notification["type"] == "reaction"
assert "emoji" in notification
assert "parent" in notification
def test_reply_notification_structure(self):
# Given: A reply notification payload
notification = {
"type": "reply",
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
"parent": "https://bob.com/social.org#2024-01-01T09:00:00+0000",
}
# Then: It carries the reply type and parent
assert notification["type"] == "reply"
assert "parent" in notification
def test_boost_notification_structure(self):
# Given: A boost notification payload
notification = {
"type": "boost",
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
"boosted": "https://bob.com/social.org#2024-01-01T09:00:00+0000",
}
# Then: It carries the boost type and the boosted post
assert notification["type"] == "boost"
assert "boosted" in notification
def test_global_notification_includes_target_feed(self):
# Given: A global stream notification payload
notification = {
"target_feed": "https://example.com/social.org",
"type": "mention",
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
}
# Then: It includes the target feed as an absolute URL
assert "target_feed" in notification
assert notification["target_feed"].startswith("http")

6
app/sse_notifications/urls.py Normal file
View file

@ -0,0 +1,6 @@
from django.urls import path
from .views import SSENotificationsView
urlpatterns = [
path("notifications/", SSENotificationsView.as_view(), name="sse-notifications"),
]

144
app/sse_notifications/views.py Normal file
View file

@ -0,0 +1,144 @@
import asyncio
import json
import time
import logging
from django.http import StreamingHttpResponse
from django.views import View
from django.conf import settings
import redis.asyncio as aioredis
logger = logging.getLogger(__name__)
HEARTBEAT_INTERVAL = 30
def _sse_response(stream):
response = StreamingHttpResponse(stream, content_type="text/event-stream")
response["Cache-Control"] = "no-cache"
response["X-Accel-Buffering"] = "no"
response["Access-Control-Allow-Origin"] = "*"
response["Access-Control-Allow-Methods"] = "GET, OPTIONS"
response["Access-Control-Allow-Headers"] = "Content-Type"
return response
def _get_redis():
host = settings.HUEY["connection"]["host"]
port = settings.HUEY["connection"]["port"]
db = settings.HUEY["connection"]["db"]
return aioredis.Redis(host=host, port=port, db=db, decode_responses=True)
class SSENotificationsView(View):
"""
Server-Sent Events endpoint for real-time notifications.
With ?feed=: streams notifications for a single feed.
Without ?feed=: streams all notifications from all feeds, adding target_feed to each event.
"""
async def get(self, request):
feed_url = request.GET.get("feed", "").strip()
stream = self._feed_stream(feed_url) if feed_url else self._global_stream()
return _sse_response(stream)
async def _feed_stream(self, feed_url):
r = _get_redis()
pubsub = r.pubsub()
logger.info(f"SSE per-feed connection: {feed_url}")
try:
await pubsub.subscribe(f"notifications:{feed_url}")
yield "event: connected\n"
yield f"data: {json.dumps({'feed': feed_url, 'status': 'connected'})}\n\n"
async for chunk in self._message_loop(pubsub):
yield chunk
except aioredis.RedisError as e:
logger.error(f"Redis error for {feed_url}: {e}")
yield "event: error\n"
yield f"data: {json.dumps({'error': 'Redis connection failed'})}\n\n"
except Exception as e:
logger.error(f"Unexpected error in SSE stream for {feed_url}: {e}")
yield "event: error\n"
yield f"data: {json.dumps({'error': 'Internal server error'})}\n\n"
finally:
try:
await pubsub.aclose()
await r.aclose()
logger.info(f"SSE connection closed for feed: {feed_url}")
except Exception:
pass
async def _global_stream(self):
r = _get_redis()
pubsub = r.pubsub()
logger.info("SSE global connection established")
try:
await pubsub.psubscribe("notifications:*")
yield "event: connected\n"
yield f"data: {json.dumps({'status': 'connected'})}\n\n"
async for chunk in self._message_loop(pubsub, global_mode=True):
yield chunk
except aioredis.RedisError as e:
logger.error(f"Redis error in global SSE stream: {e}")
yield "event: error\n"
yield f"data: {json.dumps({'error': 'Redis connection failed'})}\n\n"
except Exception as e:
logger.error(f"Unexpected error in global SSE stream: {e}")
yield "event: error\n"
yield f"data: {json.dumps({'error': 'Internal server error'})}\n\n"
finally:
try:
await pubsub.aclose()
await r.aclose()
logger.info("Global SSE connection closed")
except Exception:
pass
async def _message_loop(self, pubsub, global_mode=False):
last_heartbeat = time.time()
while True:
current_time = time.time()
if current_time - last_heartbeat >= HEARTBEAT_INTERVAL:
yield "event: heartbeat\n"
yield f"data: {json.dumps({'status': 'alive', 'timestamp': int(current_time)})}\n\n"
last_heartbeat = current_time
message = await pubsub.get_message(
ignore_subscribe_messages=True, timeout=1.0
)
if message is None:
await asyncio.sleep(0)
continue
msg_type = message["type"]
is_data = (msg_type == "message") or (
global_mode and msg_type == "pmessage"
)
if not is_data:
continue
try:
notification_data = json.loads(message["data"])
if global_mode:
channel = message["channel"]
notification_data["target_feed"] = channel.removeprefix(
"notifications:"
)
yield "event: notification\n"
yield f"data: {json.dumps(notification_data)}\n\n"
except json.JSONDecodeError as e:
logger.error(f"Failed to decode notification message: {e}")
except Exception as e:
logger.error(f"Error processing notification: {e}")

0
app/stats/__init__.py Normal file
View file

Some files were not shown because too many files have changed in this diff Show more