Compare commits

...

30 commits
v1.2 ... main

Author SHA1 Message Date
3b1926198f Document tests with Given/When/Then comments 2026-08-20 10:51:49 +02:00
fdbf8461cd Point org-social links to git.andros.dev 2026-08-20 10:24:14 +02:00
9c0e825fa1 Add License section to README 2026-08-20 10:02:19 +02:00
2728620fd6 Exclude bridge posts from global and tag-filtered RSS feeds 2026-07-28 10:05:14 +02:00
c446ae514d Exclude bridge virtual feeds from feed listing, discovery and stats
Bridges help users connect to external content (RSS, ActivityPub) but
are not real Org Social accounts. They no longer appear in /feeds/,
cannot be registered via POST, are skipped by both feed discovery
tasks, and do not count in /stats/. A data migration removes bridge
URLs already registered as feeds.
2026-07-20 09:33:28 +02:00
349a5e1376 Add ActivityPub and RSS bridges serving virtual social.org feeds
Expose external accounts as virtual Org Social feeds that any client
can follow with a plain #+FOLLOW: line:

- /bridge/activitypub/@{user}@{instance}/ bridges a Mastodon or any
  ActivityPub account (WebFinger, actor and paginated outbox; only
  public top-level notes, with CW, hashtags, language and attachments)
- /bridge/rss/?url={feed} bridges any RSS/Atom feed (entry title as
  *** sub-heading, converted body and link to the original article)

Registration is implicit on first GET. Bridged data is stored in the
existing Profile/Post tables so /profile/, /search/ and the rest of
the API work on bridged feeds. Active bridges are refreshed every 15
minutes; bridges unrequested for 90 days are cleaned up.

Remote HTML is converted to Org text, escaping headline-like lines so
external content cannot inject posts. Fetches enforce the Webmention
SSRF protections, a 10s timeout and a 5MB size cap. Bridged posts
never queue Webmentions nor publish notifications.
2026-07-17 15:26:41 +02:00
22271f2e0d Send Webmentions for external links found in posts
Implements the sender side of https://www.w3.org/TR/webmention/. During
feed scans, external URLs found in new posts (and links added by edits)
are queued as OutgoingWebmention rows; a periodic task discovers each
target's endpoint and delivers the notification. The unique
(source, target) pair guarantees a webmention is sent at most once, no
matter how many times a feed is rescanned. Targets without an endpoint
are marked permanently, failures retry with exponential backoff, and
endpoints resolving to loopback/private addresses are rejected.
Receiving webmentions is out of scope: plain text feeds cannot
advertise an endpoint.
2026-07-16 10:22:47 +02:00
f7f85153f6 Add /stats/ endpoint with monthly activity and global counters 2026-07-10 08:19:10 +02:00
25b19b948f Replace Django healthcheck with lightweight HTTP probe
manage.py check booted a full Django interpreter every 30s (~1.5s CPU
per run) and never verified the server was actually responding. Probe
the HTTP endpoint directly instead and relax the interval.
2026-07-04 19:39:30 +02:00
97ef297efe Harden feed scanning against bad birthdays and slow hosts
- Drop malformed #+BIRTHDAY values (not YYYY-MM-DD) instead of letting
  them reach the Profile DateField and abort the whole feed scan
- Use a (connect, read) timeout of (3.05, 5) when fetching feeds so dead
  hosts are dropped faster without penalizing slow-but-alive servers
- Add parser and scan_feeds regression tests
2026-06-24 09:17:21 +02:00
045e647ed2 Rewrite SSE view as async using redis.asyncio, fix nginx buffering
Sync generators blocked uvicorn's event loop preventing chunks from
being flushed. Converts to async generators with redis.asyncio so
streaming works correctly under ASGI. Adds /sse/ nginx location with
proxy_buffering off and 1h read timeout.
2026-05-19 09:18:36 +02:00
ee44472c74 Replace Django dev server with uvicorn ASGI
Switches from runserver to uvicorn for proper ASGI support, needed for
SSE streaming responses. Runs with 4 workers on port 8000.
2026-05-19 09:09:16 +02:00
0a66cfcf7a Make ?feed optional in /sse/notifications/, add global stream
Without ?feed=, the endpoint subscribes to all notification channels via
Redis psubscribe("notifications:*") and adds target_feed to each event.
With ?feed=, behavior is unchanged. Updates README and root _links.
2026-05-19 09:06:44 +02:00
347369868d Add /profile/ endpoint returning feed followers
New GET /profile/?feed={url} endpoint that returns the list of feed URLs
that follow the given profile, using the existing Follow model.
Includes cache support, 400/404 error handling, and 8 unit tests.
2026-05-16 11:58:45 +02:00
8b35862bb9 Fix cache backend race that left huey using DummyCache after restart
The previous startup probe ran a 1s TCP connect to Redis at module import
time and silently fell back to DummyCache when it failed. In docker compose
the huey container could finish importing settings.py just before Redis
finished warming up, locking the daemon to DummyCache for its entire lifetime
and breaking cache.delete/cache.clear inside scan_feeds.

Drop the probe; in DEBUG use DummyCache, otherwise always RedisCache.
2026-05-07 09:13:28 +02:00
5a6e953786 Update Contributing section with contribution guidelines link 2026-02-23 09:58:13 +01:00
Andros Fenollosa
109bb2f51c
Fix link formatting in CONTRIBUTING.md 2026-01-19 15:48:44 +01:00
Andros Fenollosa
c7edc34f7f
Create CONTRIBUTING.md 2026-01-19 15:48:32 +01:00
a47aa3ad35 Update 2026-01-19 14:59:42 +01:00
338a8b3121 Simplify diagram labels: use 'Relay' instead of 'Node'
Changed node labels from 'Node X (relay-list.txt)' to simply 'Relay X'
for cleaner visualization. Also updated Concepts section to use 'Relay'
terminology consistently throughout the documentation.
2026-01-19 14:58:52 +01:00
5483f1b7a4 Update diagram: remove centralized 'List nodes'
The relay list is now decentralized - each node maintains its own
local relay-list.txt file. Nodes share feeds directly with each other
in a P2P fashion, without needing a centralized list endpoint.

Changes:
- Remove 'List nodes' from mermaid diagram
- Add relay-list.txt notation to each node
- Update 'Share Users' to 'Share Feeds' for accuracy
- Simplify Concepts section to reflect decentralized architecture
2026-01-19 14:57:19 +01:00
17fe6a9fca Update README: relay list is now in this repository
The relay-list.txt file is now part of this repository, so the
instructions for making a relay public have been updated to reflect
that users should make PRs to this repo instead of the external
org-social repository.

When PRs are merged, all relay nodes that update their code will
automatically get the updated relay list.
2026-01-19 14:55:27 +01:00
13d80cfa0f Isolate list 2026-01-19 14:50:13 +01:00
9a18e3e958 Add comprehensive tests for feed redirect handling
Added test coverage for:
- Basic 301 redirect detection and parsing
- Feed validation with redirects
- Feed URL updates when only old URL exists
- Feed merging when both old and new URLs exist
- Mention migration without UNIQUE constraint errors

The mention migration test specifically validates the fix for the
production bug where feeds with 301 redirects were failing due to
duplicate (post, mentioned_profile) combinations.

All 23 parser tests pass successfully.
2026-01-15 11:47:42 +01:00
1d9a02c304 Fix feed redirect handling for mentions with unique constraints
The feed redirect merge logic was attempting a bulk update of mentions,
which failed when duplicate (post, mentioned_profile) combinations existed.
This caused feeds with 301 redirects to fail merging properly.

Now mentions are migrated individually, checking for existing duplicates
before updating, similar to how Follow and PollVote relationships are handled.

This fixes the UNIQUE constraint errors seen in production for feeds like:
- thesolarprincess.github.io -> thesolarprincess.site
- haiverin.scot -> www.haiverin.scot
- teoten.com -> www.teoten.com
2026-01-15 11:46:26 +01:00
4bf0d8c1d0 Remove CLAUDE.MD from git (should be local only) 2026-01-05 13:56:59 +01:00
b13cf2fad4 Add support for Org Social v1.6
New features from Org Social v1.6:
- Add LOCATION, BIRTHDAY, LANGUAGE, PINNED fields to Profile model
- Support post ID in header (** 2025-05-01T12:00:00+0100)
- Header ID takes priority over property drawer ID
- Parse and store all new v1.6 metadata fields

Changes:
- Updated Profile model with 4 new fields
- Updated parser to extract new metadata fields
- Updated parser to support ID in post headers
- Updated tasks.py to save new profile fields
- Added database migration 0010
- Added 3 new tests for v1.6 features
- Renamed SKILL.md to CLAUDE.MD

All tests passing (58/58)
2026-01-05 13:55:56 +01:00
dc4813c5fc Fix RSS feed chronological order
Parse post_id timestamp for created_at instead of using database insertion time
2025-12-25 11:10:06 +01:00
Andros Fenollosa
94d856a704 Detect and remove deleted posts during feed scanning 2025-12-11 17:04:05 +01:00
Andros Fenollosa
8ca15ffab9 Fix SSE heartbeat blocking issue
The SSE endpoint was using pubsub.listen() which blocks indefinitely
waiting for messages. This prevented heartbeats from being sent when
there was no activity, causing connections to timeout and close.

Changed to use pubsub.get_message(timeout=1) in a while loop, which
allows heartbeats to be sent every 30 seconds even when there are no
new notifications, keeping connections alive.

This fixes the issue where SSE connections would break and users
wouldn't receive real-time notifications.
2025-12-11 16:41:25 +01:00
64 changed files with 6814 additions and 431 deletions

7
CONTRIBUTING.md Normal file
View file

@ -0,0 +1,7 @@
# Contributing
Thank you for your interest in contributing to this project!
For detailed contribution guidelines, please visit:
https://git.andros.dev/andros/contribute

View file

@ -27,4 +27,4 @@ COPY . .
EXPOSE 8000 EXPOSE 8000
# Default command (can be overridden in docker-compose) # Default command (can be overridden in docker-compose)
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"] CMD ["uvicorn", "core.asgi:application", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]

330
README.md
View file

@ -2,14 +2,13 @@
## Introduction ## Introduction
Org Social Relay is a P2P system that acts as an intermediary between all [Org Social](https://github.com/tanrax/org-social) files. It scans the network, creating an index of users, mentions, replies, groups and threads. This allows you to: Org Social Relay is a P2P system that acts as an intermediary between all [Org Social](https://git.andros.dev/org-social/org-social) files. It scans the network, creating an index of users, mentions, replies, groups and threads. This allows you to:
```mermaid ```mermaid
graph TD graph TD
List["📋 List nodes"] Node1["🖥️ Relay"]
Node1["🖥️ Node 1"] Node2["🖥️ Relay"]
Node2["🖥️ Node 2"] Node3["🖥️ Relay"]
Node3["🖥️ Node 3"]
%% Social.org instances with icons %% Social.org instances with icons
Social1_1["📄 social.org"] Social1_1["📄 social.org"]
@ -19,11 +18,6 @@ graph TD
Social3_1["📄 social.org"] Social3_1["📄 social.org"]
Social3_2["📄 social.org"] Social3_2["📄 social.org"]
%% Parent-child connections with labels
List -.->|"Get"| Node1
List -.->|"Get"| Node2
List -.->|"Get"| Node3
%% Node to social.org connections %% Node to social.org connections
Social1_1 -->|"⚓ Connects"| Node1 Social1_1 -->|"⚓ Connects"| Node1
Social1_2 -->|"⚓ Connects"| Node1 Social1_2 -->|"⚓ Connects"| Node1
@ -33,19 +27,17 @@ graph TD
Social3_2 -->|"⚓ Connects"| Node3 Social3_2 -->|"⚓ Connects"| Node3
%% Bidirectional connections between nodes %% Bidirectional connections between nodes
Node1 <-.->|"👥 Share Users"| Node2 Node1 <-.->|"👥 Share Feeds"| Node2
Node2 <-.->|"👥 Share Users"| Node3 Node2 <-.->|"👥 Share Feeds"| Node3
Node1 <-.->|"👥 Share Users"| Node3 Node1 <-.->|"👥 Share Feeds"| Node3
%% Modern color scheme with gradients %% Modern color scheme with gradients
classDef socialStyle fill:#667eea,stroke:#764ba2,stroke-width:3px,color:#fff,font-weight:bold classDef socialStyle fill:#667eea,stroke:#764ba2,stroke-width:3px,color:#fff,font-weight:bold
classDef nodeStyle fill:#f093fb,stroke:#f5576c,stroke-width:3px,color:#fff,font-weight:bold classDef nodeStyle fill:#f093fb,stroke:#f5576c,stroke-width:3px,color:#fff,font-weight:bold
classDef listStyle fill:#4facfe,stroke:#00f2fe,stroke-width:4px,color:#fff,font-weight:bold
%% Apply styles %% Apply styles
class Social1_1,Social1_2,Social2_1,Social2_2,Social3_1,Social3_2 socialStyle class Social1_1,Social1_2,Social2_1,Social2_2,Social3_1,Social3_2 socialStyle
class Node1,Node2,Node3 nodeStyle class Node1,Node2,Node3 nodeStyle
class List listStyle
``` ```
[Source](/diagram.mmd) [Source](/diagram.mmd)
@ -55,12 +47,13 @@ graph TD
- Perform searches (tags and full text). - Perform searches (tags and full text).
- Participate in groups. - Participate in groups.
- See who boosted your posts. - See who boosted your posts.
- Notify external sites linked from posts via [Webmention](https://www.w3.org/TR/webmention/) (sending only; receiving them is not possible since plain text feeds cannot advertise an endpoint).
- Follow Mastodon/ActivityPub accounts and RSS/Atom feeds from any Org Social client through [bridges](#bridges-follow-activitypub-accounts-and-rss-feeds) (virtual social.org feeds).
## Concepts ## Concepts
- **List nodes**: Index of public nodes. Simple list with all the URLs of the Nodes (`https://cdn.jsdelivr.net/gh/tanrax/org-social/org-social-relay-list.txt`). It will be used by nodes to find other nodes and share information. - **Relay**: A server running Org Social Relay (this software). Each relay maintains a local `relay-list.txt` file with URLs of other relays. Relays scan the network and share feed information with each other, creating a decentralized P2P network.
- **Node**: A server running Org Social Relay (this software). It scans the network and shares information with other nodes or clients. - **Client**: An application that connects to a Relay to get information. It can be Org Social or any other application that implements the Org Social Relay API.
- **Client**: An application that connects to a Node to get information. It can be Org Social or any other application that implements the Org Social Relay API.
## Installation ## Installation
@ -93,13 +86,40 @@ nano .env
docker compose up -d docker compose up -d
``` ```
## Make your Org Social Relay public ## Managing Relay Nodes
If you want your Relay to be used by other users, and also communicate with other public Relays to work together scanning the network and improving everyone's speed, you must make a Pull Request to this file: ### Local Relay List
https://github.com/tanrax/org-social/blob/main/org-social-relay-list.txt This relay uses a local file (`relay-list.txt` in the project root) to manage the list of other relay nodes it connects to.
Add your Relay URL (e.g. `https://my-relay.example.com`) in a new line. To add or remove relay nodes from your local instance:
```bash
# Edit the relay list file
nano relay-list.txt
# Add one relay URL per line, for example:
# https://relay.org-social.org
# https://other-relay.example.com
```
The relay will automatically discover feeds from these nodes every 3 hours.
### Make your Org Social Relay public
If you want your Relay to be discoverable by other relay nodes, you can make a Pull Request to add your relay URL to the `relay-list.txt` file in this repository:
**Steps to add your relay:**
1. Fork this repository
2. Edit `relay-list.txt` and add your relay URL on a new line:
```
https://your-relay.example.com
```
3. Create a Pull Request with your changes
4. Once merged, all relay nodes that update their code (via `git pull`) will automatically discover your relay
This way, the network of relays grows organically as new nodes are added to the shared list.
## Updating ## Updating
@ -189,6 +209,7 @@ curl http://localhost:8080/
"feed-content": {"href": "/feed-content/?feed={feed_url}", "method": "GET", "templated": true}, "feed-content": {"href": "/feed-content/?feed={feed_url}", "method": "GET", "templated": true},
"notifications": {"href": "/notifications/?feed={feed_url}", "method": "GET", "templated": true}, "notifications": {"href": "/notifications/?feed={feed_url}", "method": "GET", "templated": true},
"sse-notifications": {"href": "/sse/notifications/?feed={feed_url}", "method": "GET", "templated": true}, "sse-notifications": {"href": "/sse/notifications/?feed={feed_url}", "method": "GET", "templated": true},
"sse-notifications-all": {"href": "/sse/notifications/", "method": "GET"},
"mentions": {"href": "/mentions/?feed={feed_url}", "method": "GET", "templated": true}, "mentions": {"href": "/mentions/?feed={feed_url}", "method": "GET", "templated": true},
"reactions": {"href": "/reactions/?feed={feed_url}", "method": "GET", "templated": true}, "reactions": {"href": "/reactions/?feed={feed_url}", "method": "GET", "templated": true},
"replies-to": {"href": "/replies-to/?feed={feed_url}", "method": "GET", "templated": true}, "replies-to": {"href": "/replies-to/?feed={feed_url}", "method": "GET", "templated": true},
@ -200,7 +221,11 @@ curl http://localhost:8080/
"group-messages": {"href": "/groups/{group_slug}/", "method": "GET", "templated": true}, "group-messages": {"href": "/groups/{group_slug}/", "method": "GET", "templated": true},
"polls": {"href": "/polls/", "method": "GET"}, "polls": {"href": "/polls/", "method": "GET"},
"poll-votes": {"href": "/polls/votes/?post={post_url}", "method": "GET", "templated": true}, "poll-votes": {"href": "/polls/votes/?post={post_url}", "method": "GET", "templated": true},
"rss": {"href": "/rss.xml", "method": "GET", "description": "RSS feed of latest posts (supports ?tag={tag} and ?feed={feed_url} filters)"} "profile": {"href": "/profile/?feed={feed_url}", "method": "GET", "templated": true},
"rss": {"href": "/rss.xml", "method": "GET", "description": "RSS feed of latest posts (supports ?tag={tag} and ?feed={feed_url} filters)"},
"bridge": {"href": "/bridge/", "method": "GET"},
"bridge-activitypub": {"href": "/bridge/activitypub/@{user}@{instance}/", "method": "GET", "templated": true},
"bridge-rss": {"href": "/bridge/rss/?url={feed_url}", "method": "GET", "templated": true}
} }
} }
``` ```
@ -333,10 +358,17 @@ The `by_type` breakdown in `meta` allows you to show notification counts per typ
### Real-time notifications (SSE) ### Real-time notifications (SSE)
`/sse/notifications/?feed={url feed}` - Subscribe to real-time notifications via Server-Sent Events. `/sse/notifications/` - Subscribe to real-time notifications via Server-Sent Events. The `?feed=` parameter is optional:
- **With `?feed=`**: streams only notifications for that feed.
- **Without `?feed=`**: streams all notifications from all feeds (intended for backends such as push notification services).
```sh ```sh
# Per-feed stream
curl -N "http://localhost:8080/sse/notifications/?feed=https://example.com/social.org" curl -N "http://localhost:8080/sse/notifications/?feed=https://example.com/social.org"
# Global stream (all feeds)
curl -N "http://localhost:8080/sse/notifications/"
``` ```
**Events:** **Events:**
@ -346,6 +378,11 @@ curl -N "http://localhost:8080/sse/notifications/?feed=https://example.com/socia
event: connected event: connected
data: {"feed": "https://example.com/social.org", "status": "connected"} data: {"feed": "https://example.com/social.org", "status": "connected"}
``` ```
In the global stream the `feed` field is omitted:
```
event: connected
data: {"status": "connected"}
```
- `heartbeat` - Connection keepalive (every 30s) - `heartbeat` - Connection keepalive (every 30s)
``` ```
@ -353,7 +390,7 @@ curl -N "http://localhost:8080/sse/notifications/?feed=https://example.com/socia
data: {"status": "alive", "timestamp": 1733392800} data: {"status": "alive", "timestamp": 1733392800}
``` ```
- `notification` - New notification received (same structure as `/notifications/`) - `notification` - New notification received (per-feed, same structure as `/notifications/`)
``` ```
event: notification event: notification
data: {"type": "mention", "post": "https://alice.org/social.org#2025-02-05T11:20:00+0100"} data: {"type": "mention", "post": "https://alice.org/social.org#2025-02-05T11:20:00+0100"}
@ -371,6 +408,12 @@ curl -N "http://localhost:8080/sse/notifications/?feed=https://example.com/socia
data: {"type": "boost", "post": "https://dave.org/social.org#...", "boosted": "https://example.com/social.org#..."} data: {"type": "boost", "post": "https://dave.org/social.org#...", "boosted": "https://example.com/social.org#..."}
``` ```
In the global stream, each notification includes a `target_feed` field identifying which feed received it:
```
event: notification
data: {"target_feed": "https://example.com/social.org", "type": "mention", "post": "https://alice.org/social.org#2025-02-05T11:20:00+0100"}
```
### Get mentions ### Get mentions
`/mentions/?feed={url feed}` - Get mentions for a given feed. Results are ordered from most recent to oldest. `/mentions/?feed={url feed}` - Get mentions for a given feed. Results are ordered from most recent to oldest.
@ -910,6 +953,48 @@ curl http://localhost:8080/polls/
} }
``` ```
### Get profile
`/profile/?feed={url feed}` - Get profile information for a given feed, including the list of feeds that follow it (followers).
```sh
# URL must be encoded when passed as query parameter
curl "http://localhost:8080/profile/?feed=https%3A%2F%2Fexample.com%2Fsocial.org"
# Or use curl's --data-urlencode for automatic encoding:
curl -G "http://localhost:8080/profile/" --data-urlencode "feed=https://example.com/social.org"
```
```json
{
"type": "Success",
"errors": [],
"data": {
"feed": "https://example.com/social.org",
"followers": [
"https://alice.org/social.org",
"https://bob.org/social.org",
"https://charlie.org/social.org"
]
},
"meta": {
"feed": "https://example.com/social.org",
"total_followers": 3
},
"_links": {
"self": {"href": "/profile/?feed=https%3A%2F%2Fexample.com%2Fsocial.org", "method": "GET"}
}
}
```
The response includes:
- `feed`: The queried feed URL
- `followers`: List of feed URLs that follow this feed (i.e. feeds that have included this feed in their following list)
**Error handling:**
- Returns 400 if the `feed` parameter is missing or invalid
- Returns 404 if the feed is not registered in the relay
### Get poll votes ### Get poll votes
`/polls/votes/?post={url post}` - Get votes for a specific poll. `/polls/votes/?post={url post}` - Get votes for a specific poll.
@ -962,6 +1047,183 @@ curl -G "http://localhost:8080/polls/votes/" --data-urlencode "post=https://foo.
} }
``` ```
### Get statistics
`/stats/` - Get aggregated statistics about the activity registered by the Relay. Data is grouped by year and month, plus a block of global counters. The month of each post is determined by its ID (RFC 3339 timestamp), not by when the Relay discovered it.
```sh
curl "http://localhost:8080/stats/"
```
```json
{
"type": "Success",
"errors": [],
"data": {
"years": {
"2025": {
"11": {
"active_accounts": 42,
"total_posts": 512,
"posts": 300,
"replies": 120,
"boosts": 40,
"reactions": 35,
"group_messages": 15,
"polls": 7
},
"12": {
"active_accounts": 51,
"total_posts": 640,
"posts": 380,
"replies": 150,
"boosts": 52,
"reactions": 41,
"group_messages": 22,
"polls": 4
}
},
"2026": {
"01": {
"active_accounts": 58,
"total_posts": 701,
"posts": 410,
"replies": 170,
"boosts": 60,
"reactions": 48,
"group_messages": 25,
"polls": 9
}
}
},
"global": {
"registered_feeds": 128,
"total_accounts": 115,
"total_posts": 4210,
"total_follows": 340,
"active_groups": 6
}
},
"meta": {
"generated_at": "2026-01-15T10:30:00+00:00"
},
"_links": {
"self": {"href": "/stats/", "method": "GET"},
"feeds": {"href": "/feeds/", "method": "GET"}
}
}
```
Counter definitions per month:
- `active_accounts`: Number of distinct accounts that published at least one post (of any kind) during that month.
- `total_posts`: Every post registered that month, regardless of its type.
- `posts`: Original posts, meaning posts that are neither a reply (`:REPLY_TO:`) nor a boost (`:INCLUDE:`).
- `replies`: Posts with the `:REPLY_TO:` property that contain text content. Reactions are excluded.
- `boosts`: Posts with the `:INCLUDE:` property.
- `reactions`: Replies with a `:MOOD:` property and no text content.
- `group_messages`: Posts published in any group (`:GROUP:` property).
- `polls`: Polls created that month (posts with `:POLL_END:`). Poll closings are not counted.
Global counters:
- `registered_feeds`: Total number of feeds currently registered in the Relay.
- `total_accounts`: Total number of accounts (profiles) known by the Relay.
- `total_posts`: Total number of posts registered, all time.
- `total_follows`: Total number of follow relationships between accounts.
- `active_groups`: Number of groups with at least one message.
**Note:** Counters are not mutually exclusive: a reply inside a group counts in both `replies` and `group_messages`, and a poll also counts in `posts`. Only `posts`, `replies`, `reactions` and `boosts` are disjoint between them, and together they add up to `total_posts`. Months with no activity are omitted. Like the rest of the endpoints, the response is cached and refreshed after each feed scan.
### Bridges (follow ActivityPub accounts and RSS feeds)
Bridges expose external accounts as **virtual social.org feeds** served by the relay. Any Org Social client can follow them with a plain `#+FOLLOW:` line, no client changes needed.
Registration is implicit: the first request of an unknown account fetches it from the origin and stores it. Active bridges are refreshed every 15 minutes; bridges nobody has requested for 90 days are removed automatically (and re-created on the next request).
#### ActivityPub bridge
`/bridge/activitypub/@{user}@{instance}/` - A Mastodon (or any ActivityPub) account as a virtual social.org feed (plain text).
```sh
curl http://localhost:8080/bridge/activitypub/@Mastodon@mastodon.social/
```
```org
#+TITLE: Mastodon
#+NICK: Mastodon
#+DESCRIPTION: Our mission is to connect the world...
#+AVATAR: https://files.mastodon.social/accounts/avatars/.../avatar.png
#+LINK: https://mastodon.social/@Mastodon
* Posts
** 2026-03-27T16:27:02+00:00
:PROPERTIES:
:LANG: en
:TAGS: fossback mastodon opensourcedesign ux
:END:
Following on from yesterday's blog post...
```
To follow the account, users add to their `social.org`:
```org
#+FOLLOW: https://relay.org-social.org/bridge/activitypub/@Mastodon@mastodon.social/
```
Notes:
- Handles are case-insensitive; non-canonical spellings redirect (301) to the lowercase canonical URL.
- Only public top-level posts are bridged (no replies, no boosts), like Mastodon's public RSS.
- Content warnings become a `CW:` first line; media attachments become Org links; hashtags become `:TAGS:`.
- Instances running in *authorized fetch* mode (signed requests required) cannot be bridged.
`/bridge/activitypub/` (without handle) lists the accounts already bridged by this relay.
#### RSS/Atom bridge
`/bridge/rss/?url={feed_url}` - An RSS or Atom feed as a virtual social.org feed (plain text). The `url` parameter must be URL-encoded.
```sh
curl -G http://localhost:8080/bridge/rss/ --data-urlencode "url=https://xkcd.com/rss.xml"
```
```org
#+TITLE: xkcd.com
#+NICK: xkcd_com
#+DESCRIPTION: xkcd.com: A webcomic of romance and math humor.
#+LINK: https://xkcd.com/
* Posts
** 2026-07-08T04:00:00+00:00
:PROPERTIES:
:END:
*** Airport Meeting
[[https://imgs.xkcd.com/comics/airport_meeting.png][Although it was a setback for physics...]]
[[https://xkcd.com/3269/]]
```
To follow the feed:
```org
#+FOLLOW: https://relay.org-social.org/bridge/rss/?url=https%3A%2F%2Fxkcd.com%2Frss.xml
```
Notes:
- Entry titles become `***` sub-headings inside the post, followed by the converted content and a link to the original article.
- Entries without a publication date are skipped; entries sharing the same date get IDs shifted forward one second so none is lost.
`/bridge/rss/` (without `url`) lists the feeds already bridged by this relay.
#### Bridged feeds and the rest of the API
Bridged accounts are stored as regular profiles, so the existing endpoints (`/profile/`, `/search/`, `/rss.xml?feed=...`, ...) work with the bridge URL as the `feed` parameter. Bridged posts never trigger Webmentions nor notifications: the relay only mirrors external content.
## Groups Configuration ## Groups Configuration
Org Social Relay supports organizing posts into topic-based groups. Users can join groups to participate in focused discussions. Org Social Relay supports organizing posts into topic-based groups. Users can join groups to participate in focused discussions.
@ -1028,7 +1290,7 @@ This RSS feed can be used in RSS readers to stay updated with new posts from the
## Technical information ## Technical information
You can find the public Relay list in `https://cdn.jsdelivr.net/gh/tanrax/org-social/org-social-relay-list.txt`. The relay list is stored locally in `relay-list.txt`.
### Crons ### Crons
@ -1049,3 +1311,19 @@ Every day at midnight, Relay analyzes the feeds of all registered users to disco
#### Cleanup stale feeds #### Cleanup stale feeds
Every 3 days at 2 AM, Relay automatically removes feeds that haven't been successfully fetched (HTTP 200) in the last 3 days. This keeps the relay efficient by removing inactive or dead feeds. Every 3 days at 2 AM, Relay automatically removes feeds that haven't been successfully fetched (HTTP 200) in the last 3 days. This keeps the relay efficient by removing inactive or dead feeds.
#### Refresh bridges
Every 15 minutes, Relay refetches the bridged ActivityPub accounts and RSS feeds that have been requested in the last 7 days. Dormant bridges are refreshed inline on their next request instead.
#### Cleanup stale bridges
Every 3 days at 3 AM, Relay removes bridged accounts that nobody has requested in the last 90 days, together with their stored profile and posts. They are re-created automatically if someone requests them again.
## Contributing
Contributions are welcome! Please see the [contribution guidelines](https://git.andros.dev/andros/contribute) for instructions on how to submit issues or pull requests.
## License
Org Social is free software, released under the [GPLv3 license](https://www.gnu.org/licenses/gpl-3.0.html).

0
app/bridge/__init__.py Normal file
View file

215
app/bridge/activitypub.py Normal file
View file

@ -0,0 +1,215 @@
"""
ActivityPub fetching for the bridge: WebFinger resolution, actor document
and outbox pagination, normalized into a bridge-agnostic structure.
Only public top-level notes are bridged: replies and boosts (Announce)
are skipped, mirroring what Mastodon exposes in its public RSS feeds.
Instances that require signed fetches (authorized fetch mode) will
answer 401/403 and the account cannot be bridged.
"""
import json
import logging
import re
from urllib.parse import quote
from dateutil import parser as date_parser
from .fetching import BridgeError, safe_get
from .html_to_org import html_to_org, html_to_text
logger = logging.getLogger(__name__)
HANDLE_RE = re.compile(r"^[a-z0-9._-]{1,64}@[a-z0-9-]+(\.[a-z0-9-]+)+$")
ACTIVITYPUB_ACCEPT = (
"application/activity+json, "
'application/ld+json; profile="https://www.w3.org/ns/activitystreams"'
)
MAX_POSTS = 40
MAX_OUTBOX_PAGES = 3
def normalize_handle(raw_handle):
"""
Normalize a user@instance handle: lowercase and without the optional
leading @. Returns None if the result is not a valid handle.
"""
handle = (raw_handle or "").strip().lstrip("@").lower()
if not HANDLE_RE.match(handle):
return None
return handle
def _get_json(url, description):
try:
return json.loads(safe_get(url, accept=ACTIVITYPUB_ACCEPT))
except json.JSONDecodeError as e:
raise BridgeError(f"Invalid JSON in {description} at {url}") from e
def _resolve_actor_url(user, instance):
"""Resolve the actor URL of user@instance through WebFinger."""
resource = quote(f"acct:{user}@{instance}", safe="")
webfinger_url = f"https://{instance}/.well-known/webfinger?resource={resource}"
try:
document = json.loads(safe_get(webfinger_url, accept="application/jrd+json"))
except json.JSONDecodeError as e:
raise BridgeError(f"Invalid WebFinger response from {instance}") from e
for link in document.get("links", []):
if not isinstance(link, dict) or link.get("rel") != "self":
continue
if "activity+json" in (link.get("type") or "") and link.get("href"):
return link["href"]
for link in document.get("links", []):
if isinstance(link, dict) and link.get("rel") == "self" and link.get("href"):
return link["href"]
raise BridgeError(f"No ActivityPub actor found for {user}@{instance}")
def _format_post_id(published):
"""Convert an ActivityPub published date to an Org Social post ID."""
parsed = date_parser.parse(published)
if parsed.tzinfo is None:
return None
from datetime import timezone as dt_timezone
return parsed.astimezone(dt_timezone.utc).strftime("%Y-%m-%dT%H:%M:%S+00:00")
def _note_to_post(note):
"""Normalize an ActivityPub Note object into a bridge post dict."""
published = note.get("published")
if not published:
return None
try:
post_id = _format_post_id(published)
except (ValueError, OverflowError):
logger.warning(f"Skipping note with unparseable date: {published}")
return None
if post_id is None:
return None
parts = []
summary = note.get("summary")
if summary:
parts.append(f"CW: {html_to_text(summary)}")
body = html_to_org(note.get("content") or "")
if body:
parts.append(body)
attachment_links = []
for attachment in note.get("attachment", []):
if isinstance(attachment, dict) and attachment.get("url"):
attachment_links.append(f"[[{attachment['url']}]]")
if attachment_links:
parts.append("\n".join(attachment_links))
tags = []
for tag in note.get("tag", []):
if isinstance(tag, dict) and tag.get("type") == "Hashtag" and tag.get("name"):
tags.append(tag["name"].lstrip("#"))
language = ""
content_map = note.get("contentMap")
if isinstance(content_map, dict) and content_map:
language = next(iter(content_map.keys()), "")
return {
"id": post_id,
"content": "\n\n".join(parts),
"tags": " ".join(tags),
"language": language,
}
def _collect_outbox_posts(outbox_url):
"""Walk the outbox pages collecting public top-level notes."""
document = _get_json(outbox_url, "outbox")
if "orderedItems" in document:
page = document
else:
first = document.get("first")
if isinstance(first, str):
page = _get_json(first, "outbox page")
elif isinstance(first, dict):
page = first
else:
return []
posts = []
seen_ids = set()
pages_read = 0
while page and pages_read < MAX_OUTBOX_PAGES and len(posts) < MAX_POSTS:
pages_read += 1
for item in page.get("orderedItems", []):
if len(posts) >= MAX_POSTS:
break
if not isinstance(item, dict) or item.get("type") != "Create":
continue
note = item.get("object")
if not isinstance(note, dict) or note.get("type") != "Note":
continue
if note.get("inReplyTo"):
continue
post = _note_to_post(note)
if post is None or post["id"] in seen_ids:
continue
seen_ids.add(post["id"])
posts.append(post)
next_url = page.get("next")
if isinstance(next_url, str) and len(posts) < MAX_POSTS:
page = _get_json(next_url, "outbox page")
else:
page = None
return posts
def _actor_avatar(actor):
icon = actor.get("icon")
if isinstance(icon, list):
icon = icon[0] if icon else None
if isinstance(icon, dict):
return icon.get("url") or ""
if isinstance(icon, str):
return icon
return ""
def fetch_account(user, instance):
"""
Fetch an ActivityPub account and normalize it for the bridge.
Returns a dict with actor_url, outbox_url, metadata and posts.
Raises BridgeError when the account cannot be fetched.
"""
actor_url = _resolve_actor_url(user, instance)
actor = _get_json(actor_url, "actor document")
outbox_url = actor.get("outbox")
nick = actor.get("preferredUsername") or user
metadata = {
"title": actor.get("name") or f"{nick}@{instance}",
"nick": nick,
"description": html_to_text(actor.get("summary") or ""),
"avatar": _actor_avatar(actor),
"link": actor.get("url") or actor_url,
}
posts = []
if isinstance(outbox_url, str) and outbox_url:
posts = _collect_outbox_posts(outbox_url)
else:
logger.warning(f"Actor {actor_url} has no outbox")
outbox_url = ""
return {
"actor_url": actor_url,
"outbox_url": outbox_url,
"metadata": metadata,
"posts": posts,
}

6
app/bridge/apps.py Normal file
View file

@ -0,0 +1,6 @@
from django.apps import AppConfig
class BridgeConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "app.bridge"

52
app/bridge/fetching.py Normal file
View file

@ -0,0 +1,52 @@
"""
Shared HTTP fetching helpers for the bridge app.
All bridge fetches go through safe_get(), which enforces the same SSRF
protections used for Webmentions, plus a timeout and a response size cap.
"""
import logging
import requests
from app.feeds.webmentions import is_safe_endpoint
logger = logging.getLogger(__name__)
BRIDGE_TIMEOUT = 10
BRIDGE_USER_AGENT = "Org-Social-Relay-Bridge (+https://relay.org-social.org/)"
# Maximum bytes read from any remote response
BRIDGE_MAX_BYTES = 5 * 1024 * 1024
class BridgeError(Exception):
"""A bridge source could not be fetched or understood."""
def safe_get(url, accept=None):
"""
GET a remote URL with SSRF protection, timeout and size cap.
Returns the raw bytes of the response body.
Raises BridgeError on unsafe URLs, network errors or non-2xx codes.
"""
if not is_safe_endpoint(url):
raise BridgeError(f"URL is not allowed: {url}")
headers = {"User-Agent": BRIDGE_USER_AGENT}
if accept:
headers["Accept"] = accept
try:
response = requests.get(
url, timeout=BRIDGE_TIMEOUT, headers=headers, stream=True
)
except requests.RequestException as e:
raise BridgeError(f"Could not reach {url}: {e}") from e
try:
if not (200 <= response.status_code < 300):
raise BridgeError(f"HTTP {response.status_code} from {url}")
return response.raw.read(BRIDGE_MAX_BYTES, decode_content=True)
finally:
response.close()

207
app/bridge/html_to_org.py Normal file
View file

@ -0,0 +1,207 @@
"""
Minimal HTML to Org Mode text converter for bridged content.
Converts the HTML bodies published by ActivityPub servers and RSS feeds
into Org text suitable for a post body inside a virtual social.org file.
Only the common subset is handled (paragraphs, line breaks, links,
emphasis, lists, quotes and preformatted blocks); unknown tags are
stripped, keeping their text.
"""
import re
from html.parser import HTMLParser
# A post body line must never look like an Org Social headline (* or **
# followed by whitespace) because it would break the virtual feed
# structure. Three or more asterisks are legal (post titles).
_HEADLINE_RE = re.compile(r"^(\*{1,2})(\s|$)")
_EMPHASIS_TAGS = {
"strong": "*",
"b": "*",
"em": "/",
"i": "/",
"code": "~",
"tt": "~",
}
_HEADING_TAGS = {"h1", "h2", "h3", "h4", "h5", "h6"}
def escape_org_lines(text):
"""
Prefix a space to any line that would be parsed as an Org Social
headline, so remote content cannot inject fake posts.
"""
lines = []
for line in text.split("\n"):
if _HEADLINE_RE.match(line):
line = " " + line
lines.append(line)
return "\n".join(lines)
class _OrgHTMLConverter(HTMLParser):
def __init__(self):
super().__init__(convert_charrefs=True)
self.blocks = []
self.parts = []
self.anchor_stack = [] # (href, render_as_plain_text, position)
self.emphasis_stack = [] # (marker, position)
self.in_list_item = False
self.pre_depth = 0
self.pre_parts = []
self.invisible_depth = 0
# --- block helpers -------------------------------------------------
def _flush(self):
text = "".join(self.parts)
text = re.sub(r"[ \t]+", " ", text).strip()
self.parts = []
if not text:
return
if self.in_list_item:
self.blocks.append(f"- {text}")
else:
self.blocks.append(text)
def _append_block(self, block):
self._flush()
self.blocks.append(block)
# --- parser events -------------------------------------------------
def handle_starttag(self, tag, attrs):
if self.pre_depth:
if tag == "pre":
self.pre_depth += 1
return
attrs_dict = dict(attrs)
css_classes = (attrs_dict.get("class") or "").lower()
if tag in ("p", "div"):
self._flush()
elif tag == "br":
self.parts.append("\n")
elif tag == "a":
href = (attrs_dict.get("href") or "").strip()
plain = "mention" in css_classes or "hashtag" in css_classes
self.anchor_stack.append((href, plain, len(self.parts)))
elif tag in _EMPHASIS_TAGS:
self.emphasis_stack.append((_EMPHASIS_TAGS[tag], len(self.parts)))
elif tag in ("ul", "ol"):
self._flush()
elif tag == "li":
self._flush()
self.in_list_item = True
elif tag == "blockquote":
self._append_block("#+BEGIN_QUOTE")
elif tag == "pre":
self._flush()
self.pre_depth = 1
self.pre_parts = []
elif tag in _HEADING_TAGS:
self._flush()
self.emphasis_stack.append(("*", len(self.parts)))
elif tag == "img":
src = (attrs_dict.get("src") or "").strip()
if src and "]" not in src:
alt = " ".join((attrs_dict.get("alt") or "").split())
alt = alt.replace("]", ")")
if alt:
self.parts.append(f"[[{src}][{alt}]]")
else:
self.parts.append(f"[[{src}]]")
elif tag == "span" and "invisible" in css_classes:
self.invisible_depth += 1
def handle_endtag(self, tag):
if self.pre_depth:
if tag == "pre":
self.pre_depth -= 1
if self.pre_depth == 0:
code = "".join(self.pre_parts).strip("\n")
self._append_block(f"#+BEGIN_EXAMPLE\n{code}\n#+END_EXAMPLE")
return
if tag == "a" and self.anchor_stack:
href, plain, position = self.anchor_stack.pop()
text = "".join(self.parts[position:]).strip()
if plain or not href or "]" in href or "[[" in text:
# Org links cannot nest: anchors wrapping an image keep
# only the inner image link
rendered = text
elif not text or text == href:
rendered = f"[[{href}]]"
else:
text = text.replace("]", ")")
rendered = f"[[{href}][{text}]]"
self.parts[position:] = [rendered]
elif tag in _EMPHASIS_TAGS and self.emphasis_stack:
marker, position = self.emphasis_stack.pop()
text = "".join(self.parts[position:]).strip()
self.parts[position:] = [f"{marker}{text}{marker}"] if text else []
elif tag in ("p", "div", "ul", "ol"):
self._flush()
elif tag == "li":
self._flush()
self.in_list_item = False
elif tag == "blockquote":
self._append_block("#+END_QUOTE")
elif tag in _HEADING_TAGS and self.emphasis_stack:
marker, position = self.emphasis_stack.pop()
text = "".join(self.parts[position:]).strip()
self.parts[position:] = [f"{marker}{text}{marker}"] if text else []
self._flush()
elif tag == "span" and self.invisible_depth:
self.invisible_depth -= 1
def handle_data(self, data):
if self.pre_depth:
self.pre_parts.append(data)
return
if self.invisible_depth:
return
self.parts.append(data.replace("\n", " "))
def result(self):
self._flush()
# Consecutive list items stay together; everything else is
# separated by a blank line
chunks = []
previous_is_item = False
for block in self.blocks:
is_item = block.startswith("- ")
if chunks and is_item and previous_is_item:
chunks.append("\n")
elif chunks:
chunks.append("\n\n")
chunks.append(block)
previous_is_item = is_item
return "".join(chunks)
def html_to_org(html):
"""
Convert an HTML fragment to Org text. Returns plain text with Org
links/emphasis, paragraphs separated by blank lines, and headline
injection escaped. Never raises on malformed HTML.
"""
if not html:
return ""
converter = _OrgHTMLConverter()
try:
converter.feed(html)
converter.close()
except Exception:
# html.parser is very tolerant; this is a last-resort guard
return escape_org_lines(re.sub(r"<[^>]+>", " ", html).strip())
return escape_org_lines(converter.result())
def html_to_text(html):
"""Convert an HTML fragment to a single line of plain text."""
org = html_to_org(html)
return re.sub(r"\s+", " ", org).strip()

View file

@ -0,0 +1,125 @@
# Generated by Django 6.0.7 on 2026-07-17 13:13
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = [
("feeds", "0011_outgoingwebmention"),
]
operations = [
migrations.CreateModel(
name="BridgedActivityPubAccount",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"handle",
models.CharField(
help_text="Normalized handle in user@instance form (lowercase)",
max_length=255,
unique=True,
),
),
(
"actor_url",
models.URLField(
help_text="ActivityPub actor document URL", max_length=500
),
),
(
"outbox_url",
models.URLField(
blank=True, help_text="ActivityPub outbox URL", max_length=500
),
),
(
"last_refreshed_at",
models.DateTimeField(
blank=True,
help_text="Last successful fetch from the origin",
null=True,
),
),
(
"last_accessed_at",
models.DateTimeField(
help_text="Last time a client requested this virtual feed"
),
),
("created_at", models.DateTimeField(auto_now_add=True)),
(
"profile",
models.OneToOneField(
help_text="Profile that stores the bridged data",
on_delete=django.db.models.deletion.CASCADE,
related_name="activitypub_bridge",
to="feeds.profile",
),
),
],
options={
"ordering": ["handle"],
},
),
migrations.CreateModel(
name="BridgedRssFeed",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"source_url",
models.URLField(
help_text="URL of the RSS/Atom feed",
max_length=500,
unique=True,
),
),
(
"last_refreshed_at",
models.DateTimeField(
blank=True,
help_text="Last successful fetch from the origin",
null=True,
),
),
(
"last_accessed_at",
models.DateTimeField(
help_text="Last time a client requested this virtual feed"
),
),
("created_at", models.DateTimeField(auto_now_add=True)),
(
"profile",
models.OneToOneField(
help_text="Profile that stores the bridged data",
on_delete=django.db.models.deletion.CASCADE,
related_name="rss_bridge",
to="feeds.profile",
),
),
],
options={
"ordering": ["source_url"],
},
),
]

View file

125
app/bridge/models.py Normal file
View file

@ -0,0 +1,125 @@
from urllib.parse import quote, urlparse
from django.conf import settings
from django.db import models
# Path fragments that identify a bridge virtual feed, on this relay or any other
BRIDGE_PATH_MARKERS = ("/bridge/activitypub/", "/bridge/rss/")
def is_bridge_feed_url(url):
"""
True when the URL points to a bridge virtual feed.
Bridges help users connect to external content but are not real
Org Social accounts, so they must stay out of the feed registry
and the statistics.
"""
path = urlparse(url).path
return any(marker in path for marker in BRIDGE_PATH_MARKERS)
def bridge_urls_q(field):
"""
Q object matching rows whose `field` URL belongs to a bridge virtual
feed. Use with exclude() to leave bridges out of listings and stats.
"""
q = models.Q()
for marker in BRIDGE_PATH_MARKERS:
q |= models.Q(**{f"{field}__contains": marker})
return q
def bridge_base_url():
"""
Base URL of this relay, used to build the public URL of virtual feeds.
"""
domain = settings.SITE_DOMAIN
scheme = "http" if domain.startswith(("localhost", "127.")) else "https"
return f"{scheme}://{domain}"
def activitypub_feed_url(handle):
"""Public URL of the virtual feed of a bridged ActivityPub account."""
return f"{bridge_base_url()}/bridge/activitypub/@{handle}/"
def rss_feed_url(source_url):
"""Public URL of the virtual feed of a bridged RSS feed."""
return f"{bridge_base_url()}/bridge/rss/?url={quote(source_url, safe='')}"
class BridgedActivityPubAccount(models.Model):
"""
An ActivityPub account (e.g. a Mastodon user) exposed as a virtual
Org Social feed at /bridge/activitypub/@{handle}/.
"""
handle = models.CharField(
max_length=255,
unique=True,
help_text="Normalized handle in user@instance form (lowercase)",
)
actor_url = models.URLField(
max_length=500, help_text="ActivityPub actor document URL"
)
outbox_url = models.URLField(
max_length=500, blank=True, help_text="ActivityPub outbox URL"
)
profile = models.OneToOneField(
"feeds.Profile",
on_delete=models.CASCADE,
related_name="activitypub_bridge",
help_text="Profile that stores the bridged data",
)
last_refreshed_at = models.DateTimeField(
null=True, blank=True, help_text="Last successful fetch from the origin"
)
last_accessed_at = models.DateTimeField(
help_text="Last time a client requested this virtual feed"
)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
ordering = ["handle"]
def __str__(self):
return f"@{self.handle}"
@property
def feed_url(self):
return activitypub_feed_url(self.handle)
class BridgedRssFeed(models.Model):
"""
An RSS/Atom feed exposed as a virtual Org Social feed at
/bridge/rss/?url={source_url}.
"""
source_url = models.URLField(
max_length=500, unique=True, help_text="URL of the RSS/Atom feed"
)
profile = models.OneToOneField(
"feeds.Profile",
on_delete=models.CASCADE,
related_name="rss_bridge",
help_text="Profile that stores the bridged data",
)
last_refreshed_at = models.DateTimeField(
null=True, blank=True, help_text="Last successful fetch from the origin"
)
last_accessed_at = models.DateTimeField(
help_text="Last time a client requested this virtual feed"
)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
ordering = ["source_url"]
def __str__(self):
return self.source_url
@property
def feed_url(self):
return rss_feed_url(self.source_url)

View file

@ -0,0 +1,54 @@
"""
Renders a bridged Profile and its Posts as a virtual social.org file.
The output follows the Org Social syntax so that any client can consume
the bridge URL with a plain #+FOLLOW: line.
"""
from .html_to_org import escape_org_lines
def _meta_value(value):
"""Metadata values must stay on a single line."""
return " ".join(str(value).split())
def render_profile_org(profile, posts):
"""
Build the virtual social.org content for a bridged profile.
Args:
profile: app.feeds.models.Profile instance
posts: iterable of app.feeds.models.Post, oldest first
Returns:
str: the Org Social file content
"""
lines = []
lines.append(f"#+TITLE: {_meta_value(profile.title) or profile.nick}")
lines.append(f"#+NICK: {_meta_value(profile.nick).replace(' ', '_')}")
if profile.description:
lines.append(f"#+DESCRIPTION: {_meta_value(profile.description)}")
if profile.avatar:
lines.append(f"#+AVATAR: {_meta_value(profile.avatar)}")
for link in profile.links.all():
lines.append(f"#+LINK: {_meta_value(link.url)}")
lines.append("")
lines.append("* Posts")
for post in posts:
lines.append(f"** {post.post_id}")
lines.append(":PROPERTIES:")
if post.language:
lines.append(f":LANG: {_meta_value(post.language)}")
if post.tags:
lines.append(f":TAGS: {_meta_value(post.tags)}")
lines.append(":END:")
lines.append("")
content = escape_org_lines(post.content.strip())
if content:
lines.append(content)
lines.append("")
return "\n".join(lines) + "\n"

139
app/bridge/rss.py Normal file
View file

@ -0,0 +1,139 @@
"""
RSS/Atom fetching for the bridge, normalized into the same structure
used by the ActivityPub bridge.
Each entry becomes a post whose body is the entry title as an Org
sub-heading (***), the converted entry content, and a link to the
original article.
"""
import calendar
import logging
import re
from datetime import datetime, timezone as dt_timezone
from urllib.parse import urlparse
import feedparser
from .fetching import BridgeError, safe_get
from .html_to_org import html_to_org
logger = logging.getLogger(__name__)
MAX_POSTS = 40
_NICK_ALLOWED_RE = re.compile(r"[^A-Za-z0-9_-]+")
def _derive_nick(title, source_url):
"""Build a spaces-free nick from the feed title or the feed host."""
candidate = _NICK_ALLOWED_RE.sub("_", (title or "").strip()).strip("_")
if candidate:
return candidate
host = urlparse(source_url).hostname or "feed"
return host.replace(".", "_")
def _entry_post_id(entry):
"""Org Social post ID (UTC RFC 3339) from the entry date, or None."""
parsed_time = entry.get("published_parsed") or entry.get("updated_parsed")
if not parsed_time:
return None
timestamp = calendar.timegm(parsed_time)
return datetime.fromtimestamp(timestamp, dt_timezone.utc).strftime(
"%Y-%m-%dT%H:%M:%S+00:00"
)
def _next_free_id(post_id, seen_ids):
"""
Entries sharing the same timestamp (e.g. date-only feeds) get IDs
shifted forward one second at a time so no post is lost.
"""
while post_id in seen_ids:
parsed = datetime.strptime(post_id, "%Y-%m-%dT%H:%M:%S+00:00")
shifted = parsed.timestamp() + 1
post_id = datetime.fromtimestamp(shifted, dt_timezone.utc).strftime(
"%Y-%m-%dT%H:%M:%S+00:00"
)
return post_id
def _entry_body(entry):
contents = entry.get("content") or []
if contents and contents[0].get("value"):
html = contents[0]["value"]
else:
html = entry.get("summary") or ""
return html_to_org(html)
def _entry_to_post(entry, seen_ids):
post_id = _entry_post_id(entry)
if post_id is None:
logger.warning(
f"Skipping RSS entry without date: {entry.get('link', '(no link)')}"
)
return None
post_id = _next_free_id(post_id, seen_ids)
parts = []
title = " ".join((entry.get("title") or "").split())
if title:
parts.append(f"*** {title}")
body = _entry_body(entry)
if body:
parts.append(body)
link = (entry.get("link") or "").strip()
if link and "]" not in link:
parts.append(f"[[{link}]]")
tags = " ".join(
tag.get("term", "").strip().replace(" ", "-")
for tag in entry.get("tags", [])
if tag.get("term")
)
return {
"id": post_id,
"content": "\n\n".join(parts),
"tags": tags,
"language": "",
}
def fetch_rss_feed(source_url):
"""
Fetch an RSS/Atom feed and normalize it for the bridge.
Returns a dict with metadata and posts.
Raises BridgeError when the feed cannot be fetched or parsed.
"""
raw = safe_get(source_url)
document = feedparser.parse(raw)
feed_info = document.get("feed", {})
if not document.get("entries") and not feed_info.get("title"):
raise BridgeError(f"Not a valid RSS/Atom feed: {source_url}")
title = " ".join((feed_info.get("title") or "").split())
image = feed_info.get("image") or {}
metadata = {
"title": title or source_url,
"nick": _derive_nick(title, source_url),
"description": " ".join((feed_info.get("subtitle") or "").split()),
"avatar": image.get("href", ""),
"link": feed_info.get("link") or source_url,
"language": (feed_info.get("language") or "").split("-")[0],
}
posts = []
seen_ids = set()
for entry in document.get("entries", [])[:MAX_POSTS]:
post = _entry_to_post(entry, seen_ids)
if post is None:
continue
seen_ids.add(post["id"])
posts.append(post)
return {"metadata": metadata, "posts": posts}

185
app/bridge/store.py Normal file
View file

@ -0,0 +1,185 @@
"""
Persistence and orchestration for bridged accounts.
Normalized bridge data (from activitypub.py or rss.py) is stored in the
same Profile/Post tables used for real Org Social feeds, so every
existing relay endpoint (/profile/, /search/, /rss.xml, ...) works on
bridged accounts transparently.
Bridged posts never queue Webmentions nor publish notifications: the
content belongs to external authors, the relay only mirrors it.
"""
import hashlib
import logging
from dateutil import parser as date_parser
from django.utils import timezone
from app.feeds.models import Post, Profile, ProfileLink
from . import activitypub, rss
from .models import (
BridgedActivityPubAccount,
BridgedRssFeed,
activitypub_feed_url,
rss_feed_url,
)
logger = logging.getLogger(__name__)
def store_bridge_data(feed_url, data):
"""
Create or update the Profile and Posts of a bridged account.
Args:
feed_url: public URL of the virtual feed on this relay
data: normalized dict with "metadata" and "posts"
Returns:
Profile: the stored profile
"""
metadata = data["metadata"]
posts_data = data["posts"]
content_hash = hashlib.md5(f"{metadata}{posts_data}".encode()).hexdigest()
nick = " ".join((metadata.get("nick") or "").split()).replace(" ", "_")
profile_fields = {
"title": metadata.get("title", ""),
"nick": nick,
"description": metadata.get("description", ""),
"avatar": metadata.get("avatar", ""),
"language": metadata.get("language", ""),
"version": content_hash,
}
profile, created = Profile.objects.get_or_create(
feed=feed_url, defaults=profile_fields
)
if not created:
if profile.version == content_hash:
return profile
for field, value in profile_fields.items():
setattr(profile, field, value)
profile.save()
profile.links.all().delete()
link = (metadata.get("link") or "").strip()
if link:
ProfileLink.objects.create(profile=profile, url=link)
fetched_ids = set()
oldest_created_at = None
for post_data in posts_data:
post_id = post_data["id"]
try:
post_created_at = date_parser.parse(post_id)
except (ValueError, OverflowError):
logger.warning(f"Skipping bridged post with invalid ID: {post_id}")
continue
fetched_ids.add(post_id)
if oldest_created_at is None or post_created_at < oldest_created_at:
oldest_created_at = post_created_at
post, post_created = Post.objects.get_or_create(
profile=profile,
post_id=post_id,
defaults={
"content": post_data["content"],
"tags": post_data.get("tags", "")[:500],
"language": post_data.get("language", "")[:10],
"created_at": post_created_at,
},
)
if not post_created:
post.content = post_data["content"]
post.tags = post_data.get("tags", "")[:500]
post.language = post_data.get("language", "")[:10]
post.save()
# Posts deleted at the origin disappear from the fetched window;
# remove them, but never touch posts older than what was fetched
if fetched_ids and oldest_created_at is not None:
Post.objects.filter(profile=profile, created_at__gte=oldest_created_at).exclude(
post_id__in=fetched_ids
).delete()
return profile
def create_activitypub_bridge(handle):
"""
Fetch an ActivityPub account for the first time and create its bridge.
Raises BridgeError when the account cannot be fetched.
"""
user, instance = handle.split("@", 1)
data = activitypub.fetch_account(user, instance)
profile = store_bridge_data(activitypub_feed_url(handle), data)
now = timezone.now()
bridge = BridgedActivityPubAccount.objects.create(
handle=handle,
actor_url=data["actor_url"],
outbox_url=data["outbox_url"],
profile=profile,
last_refreshed_at=now,
last_accessed_at=now,
)
logger.info(f"Created ActivityPub bridge for @{handle}")
return bridge
def refresh_activitypub_bridge(bridge):
"""
Refetch a bridged ActivityPub account and update its stored data.
Raises BridgeError when the origin cannot be fetched.
"""
user, instance = bridge.handle.split("@", 1)
data = activitypub.fetch_account(user, instance)
store_bridge_data(bridge.feed_url, data)
bridge.actor_url = data["actor_url"] or bridge.actor_url
bridge.outbox_url = data["outbox_url"] or bridge.outbox_url
bridge.last_refreshed_at = timezone.now()
bridge.save()
def create_rss_bridge(source_url):
"""
Fetch an RSS/Atom feed for the first time and create its bridge.
Raises BridgeError when the feed cannot be fetched.
"""
data = rss.fetch_rss_feed(source_url)
profile = store_bridge_data(rss_feed_url(source_url), data)
now = timezone.now()
bridge = BridgedRssFeed.objects.create(
source_url=source_url,
profile=profile,
last_refreshed_at=now,
last_accessed_at=now,
)
logger.info(f"Created RSS bridge for {source_url}")
return bridge
def refresh_rss_bridge(bridge):
"""
Refetch a bridged RSS feed and update its stored data.
Raises BridgeError when the origin cannot be fetched.
"""
data = rss.fetch_rss_feed(bridge.source_url)
store_bridge_data(bridge.feed_url, data)
bridge.last_refreshed_at = timezone.now()
bridge.save()

110
app/bridge/tasks.py Normal file
View file

@ -0,0 +1,110 @@
"""
Periodic maintenance of bridged accounts.
Only bridges requested recently are refreshed; dormant ones are
reactivated by the inline refresh on their next GET, and bridges nobody
has requested for a long time are deleted together with their profiles.
"""
import logging
from datetime import timedelta
from huey import crontab
from huey.contrib.djhuey import periodic_task
logger = logging.getLogger(__name__)
# Bridges accessed within this window are refreshed periodically
BRIDGE_ACTIVE_DAYS = 7
# Bridges not accessed for this long are deleted
BRIDGE_STALE_DAYS = 90
def _refresh_bridges_impl():
"""
Refresh every bridge accessed within the active window.
Returns:
dict: Counters of refreshed / failed bridges
"""
from django.utils import timezone
from .fetching import BridgeError
from .models import BridgedActivityPubAccount, BridgedRssFeed
from .store import refresh_activitypub_bridge, refresh_rss_bridge
cutoff = timezone.now() - timedelta(days=BRIDGE_ACTIVE_DAYS)
counters = {"refreshed": 0, "failed": 0}
refresh_plan = [
(BridgedActivityPubAccount, refresh_activitypub_bridge),
(BridgedRssFeed, refresh_rss_bridge),
]
for model, refresh in refresh_plan:
for bridge in model.objects.filter(last_accessed_at__gte=cutoff):
try:
refresh(bridge)
counters["refreshed"] += 1
except BridgeError as e:
counters["failed"] += 1
logger.warning(f"Failed to refresh bridge {bridge}: {e}")
except Exception as e:
counters["failed"] += 1
logger.error(f"Unexpected error refreshing bridge {bridge}: {e}")
logger.info(
f"Bridge refresh completed. "
f"Refreshed: {counters['refreshed']}, Failed: {counters['failed']}"
)
return counters
@periodic_task(crontab(minute="*/15")) # Run every 15 minutes
def refresh_bridges():
"""Periodic task to keep active bridged accounts up to date."""
import django
django.setup()
return _refresh_bridges_impl()
def _cleanup_stale_bridges_impl():
"""
Delete bridges (and their profiles with all posts) that nobody has
requested in BRIDGE_STALE_DAYS days.
Returns:
int: Number of bridges deleted
"""
from django.utils import timezone
from app.feeds.models import Profile
cutoff = timezone.now() - timedelta(days=BRIDGE_STALE_DAYS)
# Deleting the Profile cascades to the bridge row and its posts
stale_profiles = Profile.objects.filter(
activitypub_bridge__last_accessed_at__lt=cutoff
) | Profile.objects.filter(rss_bridge__last_accessed_at__lt=cutoff)
deleted = 0
for profile in stale_profiles:
logger.info(f"Deleting stale bridge profile: {profile.feed}")
profile.delete()
deleted += 1
if deleted:
logger.info(f"Stale bridge cleanup completed. Deleted {deleted} bridges")
return deleted
@periodic_task(crontab(day="*/3", hour=3, minute=0)) # Every 3 days at 3 AM
def cleanup_stale_bridges():
"""Periodic task to delete bridges nobody requests anymore."""
import django
django.setup()
return _cleanup_stale_bridges_impl()

View file

@ -0,0 +1,295 @@
import json
from unittest.mock import patch
from django.test import SimpleTestCase
from app.bridge.activitypub import (
MAX_POSTS,
fetch_account,
normalize_handle,
)
from app.bridge.fetching import BridgeError
WEBFINGER_URL = (
"https://mastodon.example/.well-known/webfinger"
"?resource=acct%3Aalice%40mastodon.example"
)
ACTOR_URL = "https://mastodon.example/users/alice"
OUTBOX_URL = "https://mastodon.example/users/alice/outbox"
OUTBOX_PAGE_URL = "https://mastodon.example/users/alice/outbox?page=true"
def _note_item(published, content, **extra):
note = {"type": "Note", "published": published, "content": content}
note.update(extra)
return {"type": "Create", "object": note}
def _default_responses():
return {
WEBFINGER_URL: {
"links": [
{"rel": "http://webfinger.net/rel/profile-page"},
{
"rel": "self",
"type": "application/activity+json",
"href": ACTOR_URL,
},
]
},
ACTOR_URL: {
"preferredUsername": "alice",
"name": "Alice",
"summary": "<p>My <strong>bio</strong></p>",
"icon": {"url": "https://mastodon.example/avatar.png"},
"url": "https://mastodon.example/@alice",
"outbox": OUTBOX_URL,
},
OUTBOX_URL: {"first": OUTBOX_PAGE_URL},
OUTBOX_PAGE_URL: {
"orderedItems": [
_note_item(
"2025-05-02T10:00:00Z",
"<p>Second post</p>",
tag=[{"type": "Hashtag", "name": "#emacs"}],
contentMap={"es": "<p>Segundo post</p>"},
),
_note_item(
"2025-05-01T18:00:00Z",
"<p>Reply post</p>",
inReplyTo="https://other.example/notes/1",
),
{"type": "Announce", "object": "https://other.example/notes/2"},
_note_item(
"2025-05-01T10:00:00Z",
"<p>First post</p>",
summary="Spoilers inside",
attachment=[
{
"type": "Document",
"url": "https://mastodon.example/media/cat.png",
}
],
),
]
},
}
def _fake_safe_get(responses):
def fake(url, accept=None):
if url not in responses:
raise BridgeError(f"Unexpected URL in test: {url}")
return json.dumps(responses[url]).encode()
return fake
class NormalizeHandleTest(SimpleTestCase):
"""Test cases for handle normalization."""
def test_valid_handle_is_lowercased_and_stripped(self):
# Given: A handle with leading @, capitals and spaces around
raw = " @Alice@Mastodon.Example "
# When: It is normalized
result = normalize_handle(raw)
# Then: The canonical lowercase form is returned
self.assertEqual(result, "alice@mastodon.example")
def test_handle_without_at_prefix_is_accepted(self):
# Given: A handle without the leading @
# When: It is normalized
# Then: It is valid
self.assertEqual(
normalize_handle("alice@mastodon.example"), "alice@mastodon.example"
)
def test_invalid_handles_return_none(self):
# Given: Handles missing parts or with invalid characters
invalid = [
"alice",
"alice@",
"@instance-only",
"alice@nodots",
"ali ce@mastodon.example",
"alice@mastodon.example/evil",
"",
None,
]
# When/Then: None of them normalize
for handle in invalid:
self.assertIsNone(normalize_handle(handle), handle)
class FetchAccountTest(SimpleTestCase):
"""Test cases for the ActivityPub account fetcher."""
def test_fetches_profile_metadata_from_actor(self):
# Given: A reachable account with actor metadata
responses = _default_responses()
# When: The account is fetched
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
result = fetch_account("alice", "mastodon.example")
# Then: Metadata comes from the actor document
self.assertEqual(result["actor_url"], ACTOR_URL)
self.assertEqual(result["outbox_url"], OUTBOX_URL)
self.assertEqual(result["metadata"]["title"], "Alice")
self.assertEqual(result["metadata"]["nick"], "alice")
self.assertEqual(result["metadata"]["description"], "My *bio*")
self.assertEqual(
result["metadata"]["avatar"], "https://mastodon.example/avatar.png"
)
self.assertEqual(result["metadata"]["link"], "https://mastodon.example/@alice")
def test_only_top_level_notes_become_posts(self):
# Given: An outbox with a note, a reply and an announce
responses = _default_responses()
# When: The account is fetched
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
result = fetch_account("alice", "mastodon.example")
# Then: Only the two top-level notes are bridged
self.assertEqual(len(result["posts"]), 2)
ids = [post["id"] for post in result["posts"]]
self.assertEqual(
ids, ["2025-05-02T10:00:00+00:00", "2025-05-01T10:00:00+00:00"]
)
def test_post_carries_tags_language_cw_and_attachments(self):
# Given: Notes with hashtag, contentMap, summary and attachment
responses = _default_responses()
# When: The account is fetched
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
result = fetch_account("alice", "mastodon.example")
# Then: The second post has tags and language
tagged = result["posts"][0]
self.assertEqual(tagged["tags"], "emacs")
self.assertEqual(tagged["language"], "es")
# And: The first post has the CW line and the attachment link
with_cw = result["posts"][1]
self.assertTrue(with_cw["content"].startswith("CW: Spoilers inside"))
self.assertIn("[[https://mastodon.example/media/cat.png]]", with_cw["content"])
def test_notes_without_valid_date_are_skipped(self):
# Given: An outbox with a dateless note and a naive-date note
responses = _default_responses()
responses[OUTBOX_PAGE_URL] = {
"orderedItems": [
{"type": "Create", "object": {"type": "Note", "content": "<p>x</p>"}},
_note_item("2025-05-01T10:00:00", "<p>naive date</p>"),
_note_item("2025-05-01T10:00:00Z", "<p>good</p>"),
]
}
# When: The account is fetched
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
result = fetch_account("alice", "mastodon.example")
# Then: Only the note with a timezone-aware date survives
self.assertEqual(len(result["posts"]), 1)
self.assertEqual(result["posts"][0]["content"], "good")
def test_outbox_pagination_is_followed_up_to_max_posts(self):
# Given: An outbox split into two pages with many notes
page_two_url = "https://mastodon.example/users/alice/outbox?page=2"
responses = _default_responses()
responses[OUTBOX_PAGE_URL] = {
"orderedItems": [
_note_item(f"2025-05-02T10:00:{second:02d}Z", "<p>a</p>")
for second in range(30)
],
"next": page_two_url,
}
responses[page_two_url] = {
"orderedItems": [
_note_item(f"2025-05-01T10:00:{second:02d}Z", "<p>b</p>")
for second in range(30)
]
}
# When: The account is fetched
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
result = fetch_account("alice", "mastodon.example")
# Then: Both pages are read but capped at MAX_POSTS
self.assertEqual(len(result["posts"]), MAX_POSTS)
def test_webfinger_without_actor_link_raises(self):
# Given: A WebFinger response without a self link
responses = _default_responses()
responses[WEBFINGER_URL] = {"links": [{"rel": "other"}]}
# When/Then: Fetching raises a BridgeError
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
with self.assertRaises(BridgeError):
fetch_account("alice", "mastodon.example")
def test_invalid_json_raises_bridge_error(self):
# Given: A server answering HTML instead of JSON
def fake(url, accept=None):
return b"<html>not json</html>"
# When/Then: Fetching raises a BridgeError
with patch("app.bridge.activitypub.safe_get", side_effect=fake):
with self.assertRaises(BridgeError):
fetch_account("alice", "mastodon.example")
def test_actor_without_outbox_returns_profile_without_posts(self):
# Given: An actor document without an outbox
responses = _default_responses()
del responses[ACTOR_URL]["outbox"]
# When: The account is fetched
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
result = fetch_account("alice", "mastodon.example")
# Then: The profile exists with no posts
self.assertEqual(result["posts"], [])
self.assertEqual(result["outbox_url"], "")
def test_outbox_with_inline_items_needs_no_extra_page(self):
# Given: An outbox embedding orderedItems directly
responses = _default_responses()
responses[OUTBOX_URL] = {
"orderedItems": [_note_item("2025-05-01T10:00:00Z", "<p>inline</p>")]
}
# When: The account is fetched
with patch(
"app.bridge.activitypub.safe_get",
side_effect=_fake_safe_get(responses),
):
result = fetch_account("alice", "mastodon.example")
# Then: The inline note is bridged
self.assertEqual(len(result["posts"]), 1)
self.assertEqual(result["posts"][0]["content"], "inline")

View file

@ -0,0 +1,243 @@
from django.test import SimpleTestCase
from app.bridge.html_to_org import escape_org_lines, html_to_org, html_to_text
class HtmlToOrgTest(SimpleTestCase):
"""Test cases for the HTML to Org converter."""
def test_paragraphs_become_blank_line_separated(self):
# Given: HTML with two paragraphs
html = "<p>First paragraph</p><p>Second paragraph</p>"
# When: It is converted to Org
result = html_to_org(html)
# Then: Paragraphs are separated by a blank line
self.assertEqual(result, "First paragraph\n\nSecond paragraph")
def test_br_becomes_newline(self):
# Given: HTML with a line break inside a paragraph
html = "<p>First line<br>Second line</p>"
# When: It is converted to Org
result = html_to_org(html)
# Then: The break is kept as a single newline
self.assertEqual(result, "First line\nSecond line")
def test_link_with_text_becomes_org_link(self):
# Given: An anchor with its own text
html = '<p>Read <a href="https://example.com/post">this article</a></p>'
# When: It is converted to Org
result = html_to_org(html)
# Then: An Org link with description is produced
self.assertEqual(result, "Read [[https://example.com/post][this article]]")
def test_link_with_url_as_text_becomes_plain_org_link(self):
# Given: An anchor whose text is the URL itself
html = '<a href="https://example.com">https://example.com</a>'
# When: It is converted to Org
result = html_to_org(html)
# Then: A plain Org link is produced
self.assertEqual(result, "[[https://example.com]]")
def test_hashtag_and_mention_links_stay_as_plain_text(self):
# Given: Mastodon-style hashtag and mention anchors
html = (
'<p>Hi <a href="https://m.example/@bob" class="u-url mention">'
"@bob</a> about "
'<a href="https://m.example/tags/emacs" class="mention hashtag">'
"#emacs</a></p>"
)
# When: It is converted to Org
result = html_to_org(html)
# Then: No Org links are produced, only their text
self.assertEqual(result, "Hi @bob about #emacs")
def test_emphasis_tags_become_org_markers(self):
# Given: HTML with bold, italic and code
html = "<p><strong>bold</strong> <em>italic</em> <code>code</code></p>"
# When: It is converted to Org
result = html_to_org(html)
# Then: Org emphasis markers are used
self.assertEqual(result, "*bold* /italic/ ~code~")
def test_list_items_become_org_list(self):
# Given: An unordered list
html = "<ul><li>One</li><li>Two</li></ul>"
# When: It is converted to Org
result = html_to_org(html)
# Then: Each item becomes a dash line
self.assertEqual(result, "- One\n- Two")
def test_blockquote_becomes_quote_block(self):
# Given: A quoted paragraph
html = "<blockquote><p>Wise words</p></blockquote>"
# When: It is converted to Org
result = html_to_org(html)
# Then: The text is wrapped in a quote block
self.assertEqual(result, "#+BEGIN_QUOTE\n\nWise words\n\n#+END_QUOTE")
def test_pre_becomes_example_block(self):
# Given: Preformatted content
html = "<pre>print('hi')</pre>"
# When: It is converted to Org
result = html_to_org(html)
# Then: The content is wrapped in an example block preserving text
self.assertEqual(result, "#+BEGIN_EXAMPLE\nprint('hi')\n#+END_EXAMPLE")
def test_invisible_spans_are_skipped(self):
# Given: A Mastodon-style shortened URL anchor
html = (
'<a href="https://example.com/very/long/path">'
'<span class="invisible">https://</span>'
'<span class="ellipsis">example.com/very</span></a>'
)
# When: It is converted to Org
result = html_to_org(html)
# Then: The invisible prefix is not part of the link text
self.assertEqual(
result, "[[https://example.com/very/long/path][example.com/very]]"
)
def test_image_becomes_org_link_with_alt_text(self):
# Given: An image with alt text and another without it
html = (
'<p><img src="https://example.com/a.png" alt="A comic"/></p>'
'<p><img src="https://example.com/b.png"/></p>'
)
# When: It is converted to Org
result = html_to_org(html)
# Then: Images become Org links, using alt as description
self.assertEqual(
result,
"[[https://example.com/a.png][A comic]]\n\n[[https://example.com/b.png]]",
)
def test_image_wrapped_in_anchor_keeps_only_the_image_link(self):
# Given: An image wrapped in an anchor (common in RSS feeds)
html = (
'<a href="https://example.com/post">'
'<img src="https://example.com/a.png" alt="A comic"/></a>'
)
# When: It is converted to Org
result = html_to_org(html)
# Then: No nested Org links are produced
self.assertEqual(result, "[[https://example.com/a.png][A comic]]")
def test_html_entities_are_decoded(self):
# Given: HTML with entities
html = "<p>Fish &amp; chips &lt;3</p>"
# When: It is converted to Org
result = html_to_org(html)
# Then: Entities are decoded
self.assertEqual(result, "Fish & chips <3")
def test_headline_injection_is_escaped(self):
# Given: Remote content that looks like Org Social headlines
html = "<p>** 2025-01-01T00:00:00+00:00<br>* Posts</p>"
# When: It is converted to Org
result = html_to_org(html)
# Then: The lines are prefixed so they cannot break the feed
for line in result.split("\n"):
self.assertFalse(line.startswith("*"))
def test_empty_and_missing_html_return_empty_string(self):
# Given: Empty and missing input
# When: They are converted
# Then: The result is an empty string
self.assertEqual(html_to_org(""), "")
self.assertEqual(html_to_org(None), "")
def test_unknown_tags_are_stripped_keeping_text(self):
# Given: HTML with tags the converter does not know
html = "<article><section>Some text</section></article>"
# When: It is converted to Org
result = html_to_org(html)
# Then: The text survives without the tags
self.assertEqual(result, "Some text")
class HtmlToTextTest(SimpleTestCase):
"""Test cases for the single-line text converter."""
def test_multiline_html_becomes_single_line(self):
# Given: HTML with several paragraphs
html = "<p>Bio line one</p><p>Bio line two</p>"
# When: It is converted to plain text
result = html_to_text(html)
# Then: Everything is collapsed into one line
self.assertEqual(result, "Bio line one Bio line two")
class EscapeOrgLinesTest(SimpleTestCase):
"""Test cases for headline escaping."""
def test_one_and_two_asterisk_headlines_are_escaped(self):
# Given: Text with dangerous headline-like lines
text = "* Posts\n** 2025-01-01T00:00:00+00:00\nnormal"
# When: It is escaped
result = escape_org_lines(text)
# Then: Dangerous lines are prefixed with a space
self.assertEqual(result, " * Posts\n ** 2025-01-01T00:00:00+00:00\nnormal")
def test_post_titles_with_three_asterisks_are_kept(self):
# Given: A legal Org Social post title line
text = "*** My title"
# When: It is escaped
result = escape_org_lines(text)
# Then: The title line is untouched
self.assertEqual(result, "*** My title")
def test_bold_text_at_line_start_is_kept(self):
# Given: A line starting with Org bold (no space after asterisk)
text = "*bold* rest"
# When: It is escaped
result = escape_org_lines(text)
# Then: The line is untouched
self.assertEqual(result, "*bold* rest")
def test_escaping_is_idempotent(self):
# Given: Already escaped text
text = escape_org_lines("** 2025-01-01T00:00:00+00:00")
# When: It is escaped again
result = escape_org_lines(text)
# Then: Nothing changes
self.assertEqual(result, text)

42
app/bridge/test_models.py Normal file
View file

@ -0,0 +1,42 @@
from django.test import SimpleTestCase
from app.bridge.models import is_bridge_feed_url
class IsBridgeFeedUrlTest(SimpleTestCase):
"""Test cases for is_bridge_feed_url using Given/When/Then structure."""
def test_detects_rss_bridge_url(self):
"""Test an RSS bridge URL is detected as a bridge feed."""
# Given: The URL of an RSS bridge on any relay
url = (
"https://relay.org-social.org/bridge/rss/"
"?url=https%3A%2F%2Frss.arxiv.org%2Frss%2Fquant-ph"
)
# When / Then: It is identified as a bridge feed
self.assertTrue(is_bridge_feed_url(url))
def test_detects_activitypub_bridge_url(self):
"""Test an ActivityPub bridge URL is detected as a bridge feed."""
# Given: The URL of an ActivityPub bridge on any relay
url = "https://other-relay.org/bridge/activitypub/@user@instance.tld/"
# When / Then: It is identified as a bridge feed
self.assertTrue(is_bridge_feed_url(url))
def test_regular_feed_is_not_a_bridge(self):
"""Test a regular social.org feed is not detected as a bridge."""
# Given: The URL of a real Org Social feed
url = "https://example.com/social.org"
# When / Then: It is not identified as a bridge feed
self.assertFalse(is_bridge_feed_url(url))
def test_bridge_marker_in_query_string_is_ignored(self):
"""Test the bridge marker only matches in the URL path."""
# Given: A real feed whose query string mentions a bridge path
url = "https://example.com/social.org?note=/bridge/rss/"
# When / Then: It is not identified as a bridge feed
self.assertFalse(is_bridge_feed_url(url))

View file

@ -0,0 +1,118 @@
from django.test import TestCase
from app.bridge.org_renderer import render_profile_org
from app.feeds.models import Post, Profile, ProfileLink
class RenderProfileOrgTest(TestCase):
"""Test cases for the virtual social.org renderer."""
def setUp(self):
# Given: A bridged profile with a link and two posts
self.profile = Profile.objects.create(
feed="http://localhost:8080/bridge/activitypub/@alice@m.example/",
title="Alice in Wonderland",
nick="alice",
description="Just a test account",
avatar="https://m.example/avatar.png",
version="v1",
)
ProfileLink.objects.create(profile=self.profile, url="https://m.example/@alice")
Post.objects.create(
profile=self.profile,
post_id="2025-05-01T10:00:00+00:00",
content="Hello world",
created_at="2025-05-01T10:00:00+00:00",
)
Post.objects.create(
profile=self.profile,
post_id="2025-05-02T10:00:00+00:00",
content="Second post",
tags="emacs org",
language="en",
created_at="2025-05-02T10:00:00+00:00",
)
def test_renders_global_metadata(self):
# When: The profile is rendered
result = render_profile_org(self.profile, self.profile.posts.all())
# Then: All global metadata lines are present
self.assertIn("#+TITLE: Alice in Wonderland\n", result)
self.assertIn("#+NICK: alice\n", result)
self.assertIn("#+DESCRIPTION: Just a test account\n", result)
self.assertIn("#+AVATAR: https://m.example/avatar.png\n", result)
self.assertIn("#+LINK: https://m.example/@alice\n", result)
def test_renders_posts_section_with_properties(self):
# When: The profile is rendered oldest post first
posts = self.profile.posts.order_by("created_at")
result = render_profile_org(self.profile, posts)
# Then: Posts appear under * Posts with their properties
self.assertIn("* Posts\n", result)
self.assertIn("** 2025-05-01T10:00:00+00:00\n", result)
self.assertIn("** 2025-05-02T10:00:00+00:00\n", result)
self.assertIn(":TAGS: emacs org\n", result)
self.assertIn(":LANG: en\n", result)
self.assertIn("Hello world\n", result)
# And: The first post comes before the second
self.assertLess(
result.index("2025-05-01T10:00:00+00:00"),
result.index("2025-05-02T10:00:00+00:00"),
)
def test_nick_with_spaces_is_sanitized(self):
# Given: A profile whose nick contains spaces
self.profile.nick = "alice cooper"
# When: The profile is rendered
result = render_profile_org(self.profile, [])
# Then: The nick has no spaces
self.assertIn("#+NICK: alice_cooper\n", result)
def test_metadata_never_spans_multiple_lines(self):
# Given: A profile description containing newlines
self.profile.description = "line one\nline two"
# When: The profile is rendered
result = render_profile_org(self.profile, [])
# Then: The description is collapsed into a single line
self.assertIn("#+DESCRIPTION: line one line two\n", result)
def test_post_content_headlines_are_escaped(self):
# Given: A post whose content looks like an Org Social headline
Post.objects.create(
profile=self.profile,
post_id="2025-05-03T10:00:00+00:00",
content="** 2030-01-01T00:00:00+00:00\ninjected",
created_at="2025-05-03T10:00:00+00:00",
)
# When: The profile is rendered
result = render_profile_org(
self.profile, self.profile.posts.order_by("created_at")
)
# Then: The injected headline is escaped
self.assertIn(" ** 2030-01-01T00:00:00+00:00\n", result)
self.assertNotIn("\n** 2030-01-01T00:00:00+00:00\n", result)
def test_empty_optional_metadata_is_omitted(self):
# Given: A profile with no description, avatar nor links
profile = Profile.objects.create(
feed="http://localhost:8080/bridge/rss/?url=x",
title="Bare",
nick="bare",
version="v1",
)
# When: The profile is rendered
result = render_profile_org(profile, [])
# Then: Optional lines are not present
self.assertNotIn("#+DESCRIPTION:", result)
self.assertNotIn("#+AVATAR:", result)
self.assertNotIn("#+LINK:", result)

172
app/bridge/test_rss.py Normal file
View file

@ -0,0 +1,172 @@
from unittest.mock import patch
from django.test import SimpleTestCase
from app.bridge.fetching import BridgeError
from app.bridge.rss import fetch_rss_feed
RSS_SAMPLE = b"""<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
<channel>
<title>My Blog</title>
<link>https://blog.example</link>
<description>Notes about software</description>
<language>en-us</language>
<image>
<url>https://blog.example/logo.png</url>
<title>My Blog</title>
<link>https://blog.example</link>
</image>
<item>
<title>Second article</title>
<link>https://blog.example/second</link>
<pubDate>Fri, 02 May 2025 10:00:00 GMT</pubDate>
<category>emacs</category>
<category>org mode</category>
<description>&lt;p&gt;Second &lt;strong&gt;body&lt;/strong&gt;&lt;/p&gt;</description>
</item>
<item>
<title>First article</title>
<link>https://blog.example/first</link>
<pubDate>Thu, 01 May 2025 10:00:00 GMT</pubDate>
<description>First body</description>
</item>
<item>
<title>No date article</title>
<link>https://blog.example/undated</link>
<description>Should be skipped</description>
</item>
</channel>
</rss>
"""
ATOM_SAMPLE = b"""<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>Atom Feed</title>
<subtitle>An atom feed</subtitle>
<link href="https://atom.example"/>
<entry>
<title>Atom entry</title>
<link href="https://atom.example/entry"/>
<updated>2025-05-01T10:00:00Z</updated>
<content type="html">&lt;p&gt;Atom body&lt;/p&gt;</content>
</entry>
</feed>
"""
DUPLICATED_DATES_SAMPLE = b"""<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
<channel>
<title>Daily</title>
<link>https://daily.example</link>
<item>
<title>Post A</title>
<link>https://daily.example/a</link>
<pubDate>Thu, 01 May 2025 00:00:00 GMT</pubDate>
</item>
<item>
<title>Post B</title>
<link>https://daily.example/b</link>
<pubDate>Thu, 01 May 2025 00:00:00 GMT</pubDate>
</item>
</channel>
</rss>
"""
class FetchRssFeedTest(SimpleTestCase):
"""Test cases for the RSS/Atom fetcher."""
def test_rss2_metadata_is_extracted(self):
# Given: A standard RSS 2.0 feed
with patch("app.bridge.rss.safe_get", return_value=RSS_SAMPLE):
# When: The feed is fetched
result = fetch_rss_feed("https://blog.example/feed.xml")
# Then: The channel metadata maps to profile metadata
metadata = result["metadata"]
self.assertEqual(metadata["title"], "My Blog")
self.assertEqual(metadata["nick"], "My_Blog")
self.assertEqual(metadata["description"], "Notes about software")
self.assertEqual(metadata["avatar"], "https://blog.example/logo.png")
self.assertEqual(metadata["link"], "https://blog.example")
self.assertEqual(metadata["language"], "en")
def test_entries_become_posts_with_title_body_and_link(self):
# Given: A standard RSS 2.0 feed
with patch("app.bridge.rss.safe_get", return_value=RSS_SAMPLE):
# When: The feed is fetched
result = fetch_rss_feed("https://blog.example/feed.xml")
# Then: Dated entries become posts with UTC IDs
ids = [post["id"] for post in result["posts"]]
self.assertEqual(
ids, ["2025-05-02T10:00:00+00:00", "2025-05-01T10:00:00+00:00"]
)
# And: The body has the title heading, converted content and link
second = result["posts"][0]
self.assertIn("*** Second article", second["content"])
self.assertIn("Second *body*", second["content"])
self.assertIn("[[https://blog.example/second]]", second["content"])
# And: Categories become tags with spaces replaced
self.assertEqual(second["tags"], "emacs org-mode")
def test_entries_without_date_are_skipped(self):
# Given: A feed with a dateless entry
with patch("app.bridge.rss.safe_get", return_value=RSS_SAMPLE):
# When: The feed is fetched
result = fetch_rss_feed("https://blog.example/feed.xml")
# Then: The dateless entry is not bridged
contents = " ".join(post["content"] for post in result["posts"])
self.assertNotIn("No date article", contents)
def test_atom_feeds_are_supported(self):
# Given: An Atom feed
with patch("app.bridge.rss.safe_get", return_value=ATOM_SAMPLE):
# When: The feed is fetched
result = fetch_rss_feed("https://atom.example/feed.atom")
# Then: Metadata and the entry are bridged
self.assertEqual(result["metadata"]["title"], "Atom Feed")
self.assertEqual(len(result["posts"]), 1)
self.assertEqual(result["posts"][0]["id"], "2025-05-01T10:00:00+00:00")
self.assertIn("Atom body", result["posts"][0]["content"])
def test_entries_with_same_date_get_shifted_ids(self):
# Given: A feed where two entries share the exact same date
with patch("app.bridge.rss.safe_get", return_value=DUPLICATED_DATES_SAMPLE):
# When: The feed is fetched
result = fetch_rss_feed("https://daily.example/feed.xml")
# Then: Both entries survive with distinct consecutive IDs
ids = sorted(post["id"] for post in result["posts"])
self.assertEqual(
ids, ["2025-05-01T00:00:00+00:00", "2025-05-01T00:00:01+00:00"]
)
def test_non_feed_content_raises_bridge_error(self):
# Given: A URL answering plain HTML
with patch(
"app.bridge.rss.safe_get", return_value=b"<html><body>hi</body></html>"
):
# When/Then: Fetching raises a BridgeError
with self.assertRaises(BridgeError):
fetch_rss_feed("https://blog.example/not-a-feed")
def test_feed_without_title_uses_host_as_nick(self):
# Given: A minimal feed without a channel title
sample = (
b'<?xml version="1.0"?><rss version="2.0"><channel>'
b"<item><title>x</title>"
b"<pubDate>Thu, 01 May 2025 00:00:00 GMT</pubDate></item>"
b"</channel></rss>"
)
with patch("app.bridge.rss.safe_get", return_value=sample):
# When: The feed is fetched
result = fetch_rss_feed("https://blog.example/feed.xml")
# Then: The nick is derived from the host
self.assertEqual(result["metadata"]["nick"], "blog_example")
# And: The title falls back to the source URL
self.assertEqual(result["metadata"]["title"], "https://blog.example/feed.xml")

166
app/bridge/test_store.py Normal file
View file

@ -0,0 +1,166 @@
from django.test import TestCase
from app.bridge.store import store_bridge_data
from app.feeds.models import Post, Profile
FEED_URL = "http://localhost:8080/bridge/activitypub/@alice@m.example/"
def _data(posts=None, nick="alice", title="Alice"):
return {
"metadata": {
"title": title,
"nick": nick,
"description": "Bio",
"avatar": "https://m.example/avatar.png",
"link": "https://m.example/@alice",
},
"posts": posts if posts is not None else [],
}
def _post(post_id, content, tags=""):
return {"id": post_id, "content": content, "tags": tags, "language": ""}
class StoreBridgeDataTest(TestCase):
"""Test cases for storing normalized bridge data."""
def test_creates_profile_with_posts_and_link(self):
# Given: Normalized data with two posts
data = _data(
posts=[
_post("2025-05-01T10:00:00+00:00", "First"),
_post("2025-05-02T10:00:00+00:00", "Second", tags="emacs"),
]
)
# When: The data is stored
profile = store_bridge_data(FEED_URL, data)
# Then: Profile, posts and link exist in the database
self.assertEqual(profile.feed, FEED_URL)
self.assertEqual(profile.nick, "alice")
self.assertEqual(profile.posts.count(), 2)
self.assertEqual(
list(profile.links.values_list("url", flat=True)),
["https://m.example/@alice"],
)
post = profile.posts.get(post_id="2025-05-02T10:00:00+00:00")
self.assertEqual(post.tags, "emacs")
def test_nick_with_spaces_is_sanitized(self):
# Given: Metadata with a nick containing spaces
data = _data(nick="alice cooper")
# When: The data is stored
profile = store_bridge_data(FEED_URL, data)
# Then: The nick has no spaces
self.assertEqual(profile.nick, "alice_cooper")
def test_unchanged_data_is_not_rewritten(self):
# Given: Data already stored once
data = _data(posts=[_post("2025-05-01T10:00:00+00:00", "First")])
profile = store_bridge_data(FEED_URL, data)
version = profile.version
# When: The exact same data is stored again
profile_again = store_bridge_data(FEED_URL, data)
# Then: The version hash does not change and posts are intact
self.assertEqual(profile_again.version, version)
self.assertEqual(profile_again.posts.count(), 1)
def test_changed_post_content_is_updated(self):
# Given: A stored post whose content changes at the origin
post_id = "2025-05-01T10:00:00+00:00"
store_bridge_data(FEED_URL, _data(posts=[_post(post_id, "Original")]))
# When: The new version is stored
store_bridge_data(FEED_URL, _data(posts=[_post(post_id, "Edited")]))
# Then: The post content is updated in place
profile = Profile.objects.get(feed=FEED_URL)
self.assertEqual(profile.posts.get(post_id=post_id).content, "Edited")
self.assertEqual(profile.posts.count(), 1)
def test_posts_deleted_at_origin_are_removed_within_window(self):
# Given: Three stored posts
store_bridge_data(
FEED_URL,
_data(
posts=[
_post("2025-05-01T10:00:00+00:00", "A"),
_post("2025-05-02T10:00:00+00:00", "B"),
_post("2025-05-03T10:00:00+00:00", "C"),
]
),
)
# When: The origin no longer returns the middle post
store_bridge_data(
FEED_URL,
_data(
posts=[
_post("2025-05-01T10:00:00+00:00", "A"),
_post("2025-05-03T10:00:00+00:00", "C"),
]
),
)
# Then: The deleted post is removed
profile = Profile.objects.get(feed=FEED_URL)
ids = set(profile.posts.values_list("post_id", flat=True))
self.assertEqual(
ids, {"2025-05-01T10:00:00+00:00", "2025-05-03T10:00:00+00:00"}
)
def test_posts_older_than_fetched_window_are_preserved(self):
# Given: An old post stored from a previous, deeper fetch
store_bridge_data(
FEED_URL, _data(posts=[_post("2025-01-01T10:00:00+00:00", "Old")])
)
# When: A later fetch only returns newer posts
store_bridge_data(
FEED_URL, _data(posts=[_post("2025-05-01T10:00:00+00:00", "New")])
)
# Then: The old post outside the fetched window survives
profile = Profile.objects.get(feed=FEED_URL)
ids = set(profile.posts.values_list("post_id", flat=True))
self.assertEqual(
ids, {"2025-01-01T10:00:00+00:00", "2025-05-01T10:00:00+00:00"}
)
def test_no_webmentions_are_queued_for_bridged_posts(self):
# Given: A bridged post containing an external link
from app.feeds.models import OutgoingWebmention
data = _data(
posts=[
_post(
"2025-05-01T10:00:00+00:00",
"Look at [[https://external.example/page][this]]",
)
]
)
# When: The data is stored
store_bridge_data(FEED_URL, data)
# Then: No outgoing webmention is created
self.assertEqual(OutgoingWebmention.objects.count(), 0)
def test_post_created_at_comes_from_post_id(self):
# Given: A post with a known timestamp ID
store_bridge_data(
FEED_URL, _data(posts=[_post("2025-05-01T10:00:00+00:00", "A")])
)
# When: The post is read back
post = Post.objects.get(post_id="2025-05-01T10:00:00+00:00")
# Then: created_at matches the ID timestamp
self.assertEqual(post.created_at.isoformat(), "2025-05-01T10:00:00+00:00")

160
app/bridge/test_tasks.py Normal file
View file

@ -0,0 +1,160 @@
from datetime import timedelta
from unittest.mock import patch
from django.test import TestCase
from django.utils import timezone
from app.bridge.fetching import BridgeError
from app.bridge.models import (
BridgedActivityPubAccount,
BridgedRssFeed,
activitypub_feed_url,
rss_feed_url,
)
from app.bridge.tasks import (
BRIDGE_ACTIVE_DAYS,
BRIDGE_STALE_DAYS,
_cleanup_stale_bridges_impl,
_refresh_bridges_impl,
)
from app.feeds.models import Post, Profile
def _make_ap_bridge(handle, accessed_days_ago=0):
profile = Profile.objects.create(
feed=activitypub_feed_url(handle),
title=handle,
nick=handle.split("@")[0],
version="v1",
)
return BridgedActivityPubAccount.objects.create(
handle=handle,
actor_url=f"https://{handle.split('@')[1]}/users/{handle.split('@')[0]}",
profile=profile,
last_accessed_at=timezone.now() - timedelta(days=accessed_days_ago),
)
def _make_rss_bridge(source_url, accessed_days_ago=0):
profile = Profile.objects.create(
feed=rss_feed_url(source_url),
title="Blog",
nick="blog",
version="v1",
)
return BridgedRssFeed.objects.create(
source_url=source_url,
profile=profile,
last_accessed_at=timezone.now() - timedelta(days=accessed_days_ago),
)
class RefreshBridgesTest(TestCase):
"""Test cases for the periodic bridge refresh."""
def test_only_recently_accessed_bridges_are_refreshed(self):
# Given: An active bridge and a dormant one
active = _make_ap_bridge("alice@m.example", accessed_days_ago=0)
_make_ap_bridge("bob@m.example", accessed_days_ago=BRIDGE_ACTIVE_DAYS + 1)
# When: The refresh task runs
with (
patch("app.bridge.store.refresh_activitypub_bridge") as mock_ap,
patch("app.bridge.store.refresh_rss_bridge"),
):
counters = _refresh_bridges_impl()
# Then: Only the active bridge is refreshed
self.assertEqual(counters, {"refreshed": 1, "failed": 0})
mock_ap.assert_called_once()
self.assertEqual(mock_ap.call_args[0][0].pk, active.pk)
def test_both_bridge_types_are_refreshed(self):
# Given: One active bridge of each type
_make_ap_bridge("alice@m.example")
_make_rss_bridge("https://blog.example/feed.xml")
# When: The refresh task runs
with (
patch("app.bridge.store.refresh_activitypub_bridge") as mock_ap,
patch("app.bridge.store.refresh_rss_bridge") as mock_rss,
):
counters = _refresh_bridges_impl()
# Then: Both are refreshed
self.assertEqual(counters, {"refreshed": 2, "failed": 0})
mock_ap.assert_called_once()
mock_rss.assert_called_once()
def test_failed_refreshes_are_counted_and_do_not_stop_the_task(self):
# Given: Two active bridges, the first origin failing
_make_ap_bridge("alice@m.example")
_make_ap_bridge("bob@m.example")
# When: The refresh task runs
with (
patch(
"app.bridge.store.refresh_activitypub_bridge",
side_effect=[BridgeError("down"), None],
),
patch("app.bridge.store.refresh_rss_bridge"),
):
counters = _refresh_bridges_impl()
# Then: The failure is counted and the other bridge still refreshed
self.assertEqual(counters, {"refreshed": 1, "failed": 1})
class CleanupStaleBridgesTest(TestCase):
"""Test cases for the stale bridge cleanup."""
def test_stale_bridges_are_deleted_with_profile_and_posts(self):
# Given: A stale bridge with posts and an active one
stale = _make_ap_bridge(
"old@m.example", accessed_days_ago=BRIDGE_STALE_DAYS + 1
)
Post.objects.create(
profile=stale.profile,
post_id="2025-05-01T10:00:00+00:00",
content="Bye",
created_at="2025-05-01T10:00:00+00:00",
)
active = _make_ap_bridge("new@m.example", accessed_days_ago=1)
# When: The cleanup task runs
deleted = _cleanup_stale_bridges_impl()
# Then: The stale bridge, its profile and posts are gone
self.assertEqual(deleted, 1)
self.assertFalse(BridgedActivityPubAccount.objects.filter(pk=stale.pk).exists())
self.assertFalse(
Profile.objects.filter(feed=activitypub_feed_url("old@m.example")).exists()
)
self.assertEqual(Post.objects.count(), 0)
# And: The active bridge survives
self.assertTrue(BridgedActivityPubAccount.objects.filter(pk=active.pk).exists())
def test_stale_rss_bridges_are_deleted_too(self):
# Given: A stale RSS bridge
_make_rss_bridge(
"https://blog.example/feed.xml",
accessed_days_ago=BRIDGE_STALE_DAYS + 1,
)
# When: The cleanup task runs
deleted = _cleanup_stale_bridges_impl()
# Then: It is deleted
self.assertEqual(deleted, 1)
self.assertEqual(BridgedRssFeed.objects.count(), 0)
def test_nothing_to_delete_returns_zero(self):
# Given: Only recently accessed bridges
_make_ap_bridge("alice@m.example", accessed_days_ago=1)
# When: The cleanup task runs
deleted = _cleanup_stale_bridges_impl()
# Then: Nothing is deleted
self.assertEqual(deleted, 0)
self.assertEqual(BridgedActivityPubAccount.objects.count(), 1)

383
app/bridge/test_views.py Normal file
View file

@ -0,0 +1,383 @@
from datetime import timedelta
from unittest.mock import patch
from django.test import TestCase
from django.utils import timezone
from app.bridge.fetching import BridgeError
from app.bridge.models import (
BridgedActivityPubAccount,
BridgedRssFeed,
activitypub_feed_url,
rss_feed_url,
)
from app.feeds.models import Profile
AP_FEED_PATH = "/bridge/activitypub/@alice@m.example/"
RSS_SOURCE_URL = "https://blog.example/feed.xml"
def _ap_data(posts=None):
return {
"actor_url": "https://m.example/users/alice",
"outbox_url": "https://m.example/users/alice/outbox",
"metadata": {
"title": "Alice",
"nick": "alice",
"description": "Bio",
"avatar": "https://m.example/avatar.png",
"link": "https://m.example/@alice",
},
"posts": posts
if posts is not None
else [
{
"id": "2025-05-01T10:00:00+00:00",
"content": "Hello from the fediverse",
"tags": "",
"language": "",
}
],
}
def _rss_data():
return {
"metadata": {
"title": "My Blog",
"nick": "My_Blog",
"description": "Notes",
"avatar": "",
"link": "https://blog.example",
"language": "en",
},
"posts": [
{
"id": "2025-05-01T10:00:00+00:00",
"content": "*** First article\n\nBody\n\n[[https://blog.example/first]]",
"tags": "",
"language": "",
}
],
}
class ActivityPubBridgeFeedViewTest(TestCase):
"""Test cases for the ActivityPub virtual feed endpoint."""
def test_first_get_registers_account_and_serves_org_file(self):
# Given: A reachable ActivityPub account not yet bridged
with patch(
"app.bridge.activitypub.fetch_account", return_value=_ap_data()
) as mock_fetch:
# When: The virtual feed is requested for the first time
response = self.client.get(AP_FEED_PATH)
# Then: The org file is served as plain text
self.assertEqual(response.status_code, 200)
self.assertTrue(response["Content-Type"].startswith("text/plain"))
content = response.content.decode()
self.assertIn("#+TITLE: Alice", content)
self.assertIn("#+NICK: alice", content)
self.assertIn("** 2025-05-01T10:00:00+00:00", content)
self.assertIn("Hello from the fediverse", content)
# And: The bridge and its profile are stored
mock_fetch.assert_called_once_with("alice", "m.example")
bridge = BridgedActivityPubAccount.objects.get(handle="alice@m.example")
self.assertEqual(bridge.profile.feed, activitypub_feed_url("alice@m.example"))
def test_second_get_is_served_from_database_without_network(self):
# Given: An already bridged account
with patch(
"app.bridge.activitypub.fetch_account", return_value=_ap_data()
) as mock_fetch:
self.client.get(AP_FEED_PATH)
# When: The feed is requested again shortly after
response = self.client.get(AP_FEED_PATH)
# Then: The response is correct and no second fetch happened
self.assertEqual(response.status_code, 200)
self.assertIn("Hello from the fediverse", response.content.decode())
self.assertEqual(mock_fetch.call_count, 1)
def test_non_canonical_handles_redirect_to_canonical_url(self):
# Given: Handle spellings with capitals or without the @ prefix
for path in (
"/bridge/activitypub/@Alice@M.Example/",
"/bridge/activitypub/alice@m.example/",
):
# When: The non-canonical URL is requested
response = self.client.get(path)
# Then: A permanent redirect points to the canonical URL
self.assertEqual(response.status_code, 301, path)
self.assertEqual(response["Location"], AP_FEED_PATH)
def test_invalid_handle_returns_400(self):
# Given: A handle without an instance part
# When: The feed is requested
response = self.client.get("/bridge/activitypub/@alice/")
# Then: The request is rejected
self.assertEqual(response.status_code, 400)
self.assertEqual(response.json()["type"], "Error")
def test_unreachable_account_returns_502(self):
# Given: An account whose instance cannot be reached
with patch(
"app.bridge.activitypub.fetch_account",
side_effect=BridgeError("connection refused"),
):
# When: The feed is requested
response = self.client.get(AP_FEED_PATH)
# Then: A bad gateway error with the envelope format is returned
self.assertEqual(response.status_code, 502)
body = response.json()
self.assertEqual(body["type"], "Error")
self.assertIsNone(body["data"])
# And: Nothing was stored
self.assertEqual(BridgedActivityPubAccount.objects.count(), 0)
def test_stale_bridge_is_refreshed_inline_on_access(self):
# Given: A bridged account whose data is older than a day
with patch("app.bridge.activitypub.fetch_account", return_value=_ap_data()):
self.client.get(AP_FEED_PATH)
bridge = BridgedActivityPubAccount.objects.get(handle="alice@m.example")
bridge.last_refreshed_at = timezone.now() - timedelta(days=2)
bridge.save()
new_posts = _ap_data(
posts=[
{
"id": "2025-05-05T10:00:00+00:00",
"content": "Fresh post",
"tags": "",
"language": "",
}
]
)
# When: The feed is requested again
with patch(
"app.bridge.activitypub.fetch_account", return_value=new_posts
) as mock_fetch:
response = self.client.get(AP_FEED_PATH)
# Then: The origin was fetched again and the new post is served
mock_fetch.assert_called_once()
self.assertIn("Fresh post", response.content.decode())
bridge.refresh_from_db()
self.assertGreater(
bridge.last_refreshed_at, timezone.now() - timedelta(minutes=1)
)
def test_failed_inline_refresh_serves_stored_copy(self):
# Given: A stale bridge whose origin is now unreachable
with patch("app.bridge.activitypub.fetch_account", return_value=_ap_data()):
self.client.get(AP_FEED_PATH)
bridge = BridgedActivityPubAccount.objects.get(handle="alice@m.example")
bridge.last_refreshed_at = timezone.now() - timedelta(days=2)
bridge.save()
# When: The feed is requested while the origin fails
with patch(
"app.bridge.activitypub.fetch_account",
side_effect=BridgeError("down"),
):
response = self.client.get(AP_FEED_PATH)
# Then: The stored copy is served anyway
self.assertEqual(response.status_code, 200)
self.assertIn("Hello from the fediverse", response.content.decode())
def test_bridged_profile_is_available_through_profile_endpoint(self):
# Given: A bridged account
with patch("app.bridge.activitypub.fetch_account", return_value=_ap_data()):
self.client.get(AP_FEED_PATH)
# When: The existing /profile/ endpoint is queried with the bridge URL
feed_url = activitypub_feed_url("alice@m.example")
response = self.client.get("/profile/", {"feed": feed_url})
# Then: The bridged profile is returned like any other profile
self.assertEqual(response.status_code, 200)
self.assertEqual(response.json()["type"], "Success")
class ActivityPubBridgeListViewTest(TestCase):
"""Test cases for the bridged accounts listing."""
def test_lists_bridged_accounts_with_feed_urls(self):
# Given: A bridged account
profile = Profile.objects.create(
feed=activitypub_feed_url("alice@m.example"),
title="Alice",
nick="alice",
version="v1",
)
BridgedActivityPubAccount.objects.create(
handle="alice@m.example",
actor_url="https://m.example/users/alice",
profile=profile,
last_accessed_at=timezone.now(),
)
# When: The list is requested
response = self.client.get("/bridge/activitypub/")
# Then: The account and its virtual feed URL are listed
self.assertEqual(response.status_code, 200)
body = response.json()
self.assertEqual(body["type"], "Success")
self.assertEqual(
body["data"],
[
{
"handle": "@alice@m.example",
"feed": activitypub_feed_url("alice@m.example"),
}
],
)
class RssBridgeViewTest(TestCase):
"""Test cases for the RSS bridge endpoint."""
def test_first_get_registers_feed_and_serves_org_file(self):
# Given: A reachable RSS feed not yet bridged
with patch(
"app.bridge.rss.fetch_rss_feed", return_value=_rss_data()
) as mock_fetch:
# When: The virtual feed is requested for the first time
response = self.client.get("/bridge/rss/", {"url": RSS_SOURCE_URL})
# Then: The org file is served as plain text
self.assertEqual(response.status_code, 200)
self.assertTrue(response["Content-Type"].startswith("text/plain"))
content = response.content.decode()
self.assertIn("#+TITLE: My Blog", content)
self.assertIn("#+NICK: My_Blog", content)
self.assertIn("*** First article", content)
# And: The bridge is stored
mock_fetch.assert_called_once_with(RSS_SOURCE_URL)
self.assertTrue(
BridgedRssFeed.objects.filter(source_url=RSS_SOURCE_URL).exists()
)
def test_second_get_is_served_from_database_without_network(self):
# Given: An already bridged RSS feed
with patch(
"app.bridge.rss.fetch_rss_feed", return_value=_rss_data()
) as mock_fetch:
self.client.get("/bridge/rss/", {"url": RSS_SOURCE_URL})
# When: The feed is requested again shortly after
response = self.client.get("/bridge/rss/", {"url": RSS_SOURCE_URL})
# Then: The response is correct and no second fetch happened
self.assertEqual(response.status_code, 200)
self.assertEqual(mock_fetch.call_count, 1)
def test_without_url_parameter_lists_bridged_feeds(self):
# Given: A bridged RSS feed
profile = Profile.objects.create(
feed=rss_feed_url(RSS_SOURCE_URL),
title="My Blog",
nick="My_Blog",
version="v1",
)
BridgedRssFeed.objects.create(
source_url=RSS_SOURCE_URL,
profile=profile,
last_accessed_at=timezone.now(),
)
# When: The endpoint is requested without url
response = self.client.get("/bridge/rss/")
# Then: The bridged feeds are listed
self.assertEqual(response.status_code, 200)
body = response.json()
self.assertEqual(
body["data"],
[{"url": RSS_SOURCE_URL, "feed": rss_feed_url(RSS_SOURCE_URL)}],
)
def test_invalid_url_parameter_returns_400(self):
# Given: Invalid url parameters
too_long = "https://example.com/" + "a" * 500
for bad_url in ("ftp://example.com/feed", "not-a-url", too_long):
# When: The endpoint is requested
response = self.client.get("/bridge/rss/", {"url": bad_url})
# Then: The request is rejected
self.assertEqual(response.status_code, 400, bad_url)
self.assertEqual(response.json()["type"], "Error")
def test_unreachable_feed_returns_502(self):
# Given: A feed URL that cannot be fetched
with patch(
"app.bridge.rss.fetch_rss_feed",
side_effect=BridgeError("timeout"),
):
# When: The endpoint is requested
response = self.client.get("/bridge/rss/", {"url": RSS_SOURCE_URL})
# Then: A bad gateway error is returned and nothing is stored
self.assertEqual(response.status_code, 502)
self.assertEqual(BridgedRssFeed.objects.count(), 0)
class BridgeIndexViewTest(TestCase):
"""Test cases for the bridge index endpoint."""
def test_index_exposes_bridge_links(self):
# When: The bridge index is requested
response = self.client.get("/bridge/")
# Then: Both bridge types are discoverable
self.assertEqual(response.status_code, 200)
links = response.json()["_links"]
self.assertIn("activitypub-feed", links)
self.assertIn("rss-feed", links)
def test_root_endpoint_links_to_bridges(self):
# When: The relay root is requested
response = self.client.get("/")
# Then: The bridge endpoints are listed
links = response.json()["_links"]
self.assertIn("bridge", links)
self.assertIn("bridge-activitypub", links)
self.assertIn("bridge-rss", links)
class LastAccessedTrackingTest(TestCase):
"""Test cases for access tracking used by refresh and cleanup."""
def test_old_last_accessed_is_updated_on_get(self):
# Given: A bridge last accessed two days ago
profile = Profile.objects.create(
feed=activitypub_feed_url("alice@m.example"),
title="Alice",
nick="alice",
version="v1",
)
old_access = timezone.now() - timedelta(days=2)
bridge = BridgedActivityPubAccount.objects.create(
handle="alice@m.example",
actor_url="https://m.example/users/alice",
profile=profile,
last_accessed_at=old_access,
last_refreshed_at=timezone.now(),
)
# When: The feed is requested
self.client.get(AP_FEED_PATH)
# Then: last_accessed_at moves forward
bridge.refresh_from_db()
self.assertGreater(bridge.last_accessed_at, old_access)

23
app/bridge/urls.py Normal file
View file

@ -0,0 +1,23 @@
from django.urls import path, re_path
from .views import (
ActivityPubBridgeFeedView,
ActivityPubBridgeListView,
BridgeIndexView,
RssBridgeView,
)
urlpatterns = [
path("", BridgeIndexView.as_view(), name="bridge-index"),
path(
"activitypub/",
ActivityPubBridgeListView.as_view(),
name="bridge-activitypub-list",
),
re_path(
r"^activitypub/(?P<handle>[^/]+)/$",
ActivityPubBridgeFeedView.as_view(),
name="bridge-activitypub-feed",
),
path("rss/", RssBridgeView.as_view(), name="bridge-rss"),
]

207
app/bridge/views.py Normal file
View file

@ -0,0 +1,207 @@
"""
Bridge endpoints: expose ActivityPub accounts and RSS/Atom feeds as
virtual social.org files that any Org Social client can #+FOLLOW:.
Registration is implicit: the first GET of an unknown account fetches
it from the origin and stores it. Later requests are served from the
database; the periodic refresh task keeps active bridges up to date,
and a request older than REFRESH_ON_ACCESS triggers an inline refresh.
"""
import logging
from datetime import timedelta
from django.http import HttpResponse, HttpResponsePermanentRedirect
from django.utils import timezone
from rest_framework import status
from rest_framework.response import Response
from rest_framework.views import APIView
from . import store
from .activitypub import normalize_handle
from .fetching import BridgeError
from .models import BridgedActivityPubAccount, BridgedRssFeed
from .org_renderer import render_profile_org
logger = logging.getLogger(__name__)
# A GET on data older than this triggers a synchronous refresh, which
# also reactivates bridges that fell out of the periodic refresh window
REFRESH_ON_ACCESS = timedelta(hours=24)
# last_accessed_at is written at most once per hour
ACCESS_UPDATE_THROTTLE = timedelta(hours=1)
MAX_SOURCE_URL_LENGTH = 500
def _error_response(errors, http_status):
return Response(
{"type": "Error", "errors": errors, "data": None}, status=http_status
)
def _serve_bridge(bridge, refresh_callable):
"""Common flow: refresh if stale, track access, render the org file."""
now = timezone.now()
if (
bridge.last_refreshed_at is None
or now - bridge.last_refreshed_at > REFRESH_ON_ACCESS
):
try:
refresh_callable(bridge)
except BridgeError as e:
# Serve the stored copy; the origin may recover later
logger.warning(f"Inline refresh failed for {bridge}: {e}")
if now - bridge.last_accessed_at > ACCESS_UPDATE_THROTTLE:
bridge.last_accessed_at = now
bridge.save(update_fields=["last_accessed_at"])
posts = bridge.profile.posts.order_by("created_at")
content = render_profile_org(bridge.profile, posts)
return HttpResponse(content, content_type="text/plain; charset=utf-8")
class BridgeIndexView(APIView):
"""Information about the available bridges."""
def get(self, request):
return Response(
{
"type": "Success",
"errors": [],
"data": {
"description": (
"Bridges expose external accounts as virtual "
"social.org feeds that can be followed with #+FOLLOW:"
),
},
"_links": {
"self": {"href": "/bridge/", "method": "GET"},
"activitypub-feed": {
"href": "/bridge/activitypub/@{user}@{instance}/",
"method": "GET",
"templated": True,
},
"activitypub-list": {
"href": "/bridge/activitypub/",
"method": "GET",
},
"rss-feed": {
"href": "/bridge/rss/?url={feed_url}",
"method": "GET",
"templated": True,
},
"rss-list": {"href": "/bridge/rss/", "method": "GET"},
},
}
)
class ActivityPubBridgeListView(APIView):
"""List bridged ActivityPub accounts."""
def get(self, request):
data = [
{"handle": f"@{bridge.handle}", "feed": bridge.feed_url}
for bridge in BridgedActivityPubAccount.objects.all()
]
return Response(
{
"type": "Success",
"errors": [],
"data": data,
"_links": {
"self": {"href": "/bridge/activitypub/", "method": "GET"},
},
}
)
class ActivityPubBridgeFeedView(APIView):
"""Serve an ActivityPub account as a virtual social.org file."""
def get(self, request, handle):
normalized = normalize_handle(handle)
if normalized is None:
return _error_response(
["Invalid handle. Expected format: @user@instance"],
status.HTTP_400_BAD_REQUEST,
)
# Redirect non-canonical spellings so each account has one URL
if handle != f"@{normalized}":
return HttpResponsePermanentRedirect(f"/bridge/activitypub/@{normalized}/")
bridge = (
BridgedActivityPubAccount.objects.select_related("profile")
.filter(handle=normalized)
.first()
)
if bridge is None:
try:
bridge = store.create_activitypub_bridge(normalized)
except BridgeError as e:
logger.warning(f"Could not bridge @{normalized}: {e}")
return _error_response(
[f"Could not fetch ActivityPub account: {e}"],
status.HTTP_502_BAD_GATEWAY,
)
return _serve_bridge(bridge, store.refresh_activitypub_bridge)
class RssBridgeView(APIView):
"""
With ?url= serves an RSS/Atom feed as a virtual social.org file;
without it, lists the bridged RSS feeds.
"""
def get(self, request):
source_url = request.query_params.get("url")
if source_url is None:
data = [
{"url": bridge.source_url, "feed": bridge.feed_url}
for bridge in BridgedRssFeed.objects.all()
]
return Response(
{
"type": "Success",
"errors": [],
"data": data,
"_links": {
"self": {"href": "/bridge/rss/", "method": "GET"},
},
}
)
source_url = source_url.strip()
if (
not source_url.startswith(("http://", "https://"))
or len(source_url) > MAX_SOURCE_URL_LENGTH
):
return _error_response(
["Invalid url parameter. Expected an http(s) feed URL"],
status.HTTP_400_BAD_REQUEST,
)
bridge = (
BridgedRssFeed.objects.select_related("profile")
.filter(source_url=source_url)
.first()
)
if bridge is None:
try:
bridge = store.create_rss_bridge(source_url)
except BridgeError as e:
logger.warning(f"Could not bridge RSS feed {source_url}: {e}")
return _error_response(
[f"Could not fetch RSS feed: {e}"],
status.HTTP_502_BAD_GATEWAY,
)
return _serve_bridge(bridge, store.refresh_rss_bridge)

View file

@ -7,8 +7,8 @@ import requests
class FeedContentViewTests(TestCase): class FeedContentViewTests(TestCase):
def setUp(self): def setUp(self):
"""Set up test fixtures."""
self.client = APIClient() self.client = APIClient()
# Create test profile
self.profile = Profile.objects.create( self.profile = Profile.objects.create(
feed="https://example.com/social.org", feed="https://example.com/social.org",
title="Test Profile", title="Test Profile",
@ -17,6 +17,7 @@ class FeedContentViewTests(TestCase):
def test_get_feed_content_success(self): def test_get_feed_content_success(self):
"""Test successfully fetching feed content""" """Test successfully fetching feed content"""
# Given: A registered feed whose server returns valid Org Social content
mock_content = """#+TITLE: My Social Feed mock_content = """#+TITLE: My Social Feed
#+AUTHOR: John Doe #+AUTHOR: John Doe
@ -35,10 +36,12 @@ Hello, world! This is my first post.
mock_response.raise_for_status = Mock() mock_response.raise_for_status = Mock()
mock_get.return_value = mock_response mock_get.return_value = mock_response
# When: The feed content endpoint is requested
response = self.client.get( response = self.client.get(
"/feed-content/", {"feed": "https://example.com/social.org"} "/feed-content/", {"feed": "https://example.com/social.org"}
) )
# Then: The content is returned verbatim with HATEOAS links
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
self.assertEqual(response.data["type"], "Success") self.assertEqual(response.data["type"], "Success")
self.assertEqual(response.data["data"]["content"], mock_content) self.assertEqual(response.data["data"]["content"], mock_content)
@ -47,58 +50,77 @@ Hello, world! This is my first post.
def test_get_feed_content_missing_parameter(self): def test_get_feed_content_missing_parameter(self):
"""Test error when feed parameter is missing""" """Test error when feed parameter is missing"""
# Given: No feed parameter
# When: The feed content endpoint is requested
response = self.client.get("/feed-content/") response = self.client.get("/feed-content/")
# Then: A 400 error explains the missing parameter
self.assertEqual(response.status_code, 400) self.assertEqual(response.status_code, 400)
self.assertEqual(response.data["type"], "Error") self.assertEqual(response.data["type"], "Error")
self.assertIn("Feed URL parameter is required", response.data["errors"]) self.assertIn("Feed URL parameter is required", response.data["errors"])
def test_get_feed_content_empty_parameter(self): def test_get_feed_content_empty_parameter(self):
"""Test error when feed parameter is empty""" """Test error when feed parameter is empty"""
# Given: A blank feed parameter
# When: The feed content endpoint is requested
response = self.client.get("/feed-content/", {"feed": " "}) response = self.client.get("/feed-content/", {"feed": " "})
# Then: A 400 error explains the missing parameter
self.assertEqual(response.status_code, 400) self.assertEqual(response.status_code, 400)
self.assertEqual(response.data["type"], "Error") self.assertEqual(response.data["type"], "Error")
self.assertIn("Feed URL parameter is required", response.data["errors"]) self.assertIn("Feed URL parameter is required", response.data["errors"])
def test_get_feed_content_feed_not_found(self): def test_get_feed_content_feed_not_found(self):
"""Test error when feed is not registered in relay""" """Test error when feed is not registered in relay"""
# Given: A feed URL that is not registered in the relay
# When: The feed content endpoint is requested
response = self.client.get( response = self.client.get(
"/feed-content/", {"feed": "https://unknown.com/social.org"} "/feed-content/", {"feed": "https://unknown.com/social.org"}
) )
# Then: A 404 error explains the feed is unknown
self.assertEqual(response.status_code, 404) self.assertEqual(response.status_code, 404)
self.assertEqual(response.data["type"], "Error") self.assertEqual(response.data["type"], "Error")
self.assertIn("Feed not found in relay", response.data["errors"]) self.assertIn("Feed not found in relay", response.data["errors"])
def test_get_feed_content_timeout(self): def test_get_feed_content_timeout(self):
"""Test error when feed server times out""" """Test error when feed server times out"""
# Given: A registered feed whose server times out
with patch("app.feedcontent.views.requests.get") as mock_get: with patch("app.feedcontent.views.requests.get") as mock_get:
mock_get.side_effect = requests.exceptions.Timeout() mock_get.side_effect = requests.exceptions.Timeout()
# When: The feed content endpoint is requested
response = self.client.get( response = self.client.get(
"/feed-content/", {"feed": "https://example.com/social.org"} "/feed-content/", {"feed": "https://example.com/social.org"}
) )
# Then: A 502 error reports the timeout
self.assertEqual(response.status_code, 502) self.assertEqual(response.status_code, 502)
self.assertEqual(response.data["type"], "Error") self.assertEqual(response.data["type"], "Error")
self.assertIn("Request timeout", response.data["errors"][0]) self.assertIn("Request timeout", response.data["errors"][0])
def test_get_feed_content_connection_error(self): def test_get_feed_content_connection_error(self):
"""Test error when connection to feed server fails""" """Test error when connection to feed server fails"""
# Given: A registered feed whose server refuses the connection
with patch("app.feedcontent.views.requests.get") as mock_get: with patch("app.feedcontent.views.requests.get") as mock_get:
mock_get.side_effect = requests.exceptions.ConnectionError() mock_get.side_effect = requests.exceptions.ConnectionError()
# When: The feed content endpoint is requested
response = self.client.get( response = self.client.get(
"/feed-content/", {"feed": "https://example.com/social.org"} "/feed-content/", {"feed": "https://example.com/social.org"}
) )
# Then: A 502 error reports the connection failure
self.assertEqual(response.status_code, 502) self.assertEqual(response.status_code, 502)
self.assertEqual(response.data["type"], "Error") self.assertEqual(response.data["type"], "Error")
self.assertIn("Connection error", response.data["errors"][0]) self.assertIn("Connection error", response.data["errors"][0])
def test_get_feed_content_http_error(self): def test_get_feed_content_http_error(self):
"""Test error when feed server returns HTTP error""" """Test error when feed server returns HTTP error"""
# Given: A registered feed whose server answers with an HTTP error
with patch("app.feedcontent.views.requests.get") as mock_get: with patch("app.feedcontent.views.requests.get") as mock_get:
mock_response = Mock() mock_response = Mock()
mock_response.raise_for_status.side_effect = requests.exceptions.HTTPError( mock_response.raise_for_status.side_effect = requests.exceptions.HTTPError(
@ -106,16 +128,19 @@ Hello, world! This is my first post.
) )
mock_get.return_value = mock_response mock_get.return_value = mock_response
# When: The feed content endpoint is requested
response = self.client.get( response = self.client.get(
"/feed-content/", {"feed": "https://example.com/social.org"} "/feed-content/", {"feed": "https://example.com/social.org"}
) )
# Then: A 502 error reports the upstream HTTP error
self.assertEqual(response.status_code, 502) self.assertEqual(response.status_code, 502)
self.assertEqual(response.data["type"], "Error") self.assertEqual(response.data["type"], "Error")
self.assertIn("HTTP error", response.data["errors"][0]) self.assertIn("HTTP error", response.data["errors"][0])
def test_get_feed_content_unicode(self): def test_get_feed_content_unicode(self):
"""Test fetching feed content with unicode characters""" """Test fetching feed content with unicode characters"""
# Given: A registered feed whose content includes unicode and emojis
mock_content = """#+TITLE: Mi Feed Social mock_content = """#+TITLE: Mi Feed Social
#+AUTHOR: José García #+AUTHOR: José García
@ -134,10 +159,12 @@ Hello, world! This is my first post.
mock_response.raise_for_status = Mock() mock_response.raise_for_status = Mock()
mock_get.return_value = mock_response mock_get.return_value = mock_response
# When: The feed content endpoint is requested
response = self.client.get( response = self.client.get(
"/feed-content/", {"feed": "https://example.com/social.org"} "/feed-content/", {"feed": "https://example.com/social.org"}
) )
# Then: The unicode content survives the round trip intact
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
self.assertEqual(response.data["type"], "Success") self.assertEqual(response.data["type"], "Success")
self.assertEqual(response.data["data"]["content"], mock_content) self.assertEqual(response.data["data"]["content"], mock_content)
@ -146,6 +173,7 @@ Hello, world! This is my first post.
def test_get_feed_content_preserves_formatting(self): def test_get_feed_content_preserves_formatting(self):
"""Test that feed content preserves whitespace and formatting""" """Test that feed content preserves whitespace and formatting"""
# Given: A registered feed whose content mixes blank lines, spaces and tabs
mock_content = """#+TITLE: Test Feed mock_content = """#+TITLE: Test Feed
* 2025-02-05T10:00:00+0100 * 2025-02-05T10:00:00+0100
@ -166,10 +194,11 @@ Line 2 with multiple spaces
mock_response.raise_for_status = Mock() mock_response.raise_for_status = Mock()
mock_get.return_value = mock_response mock_get.return_value = mock_response
# When: The feed content endpoint is requested
response = self.client.get( response = self.client.get(
"/feed-content/", {"feed": "https://example.com/social.org"} "/feed-content/", {"feed": "https://example.com/social.org"}
) )
# Then: The content is exactly as provided, preserving all whitespace
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
# Content should be exactly as provided, preserving all whitespace
self.assertEqual(response.data["data"]["content"], mock_content) self.assertEqual(response.data["data"]["content"], mock_content)

View file

@ -0,0 +1,19 @@
# Generated by Django 5.2.6 on 2025-12-25 09:01
import django.utils.timezone
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('feeds', '0008_post_include'),
]
operations = [
migrations.AlterField(
model_name='post',
name='created_at',
field=models.DateTimeField(default=django.utils.timezone.now, help_text='Creation timestamp, parsed from post_id when available'),
),
]

View file

@ -0,0 +1,33 @@
# Generated by Django 5.2.6 on 2026-01-05 12:49
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('feeds', '0009_alter_post_created_at'),
]
operations = [
migrations.AddField(
model_name='profile',
name='birthday',
field=models.DateField(blank=True, help_text='User birthday in YYYY-MM-DD format', null=True),
),
migrations.AddField(
model_name='profile',
name='language',
field=models.CharField(blank=True, help_text='Space-separated language codes (ISO 639-1)', max_length=100),
),
migrations.AddField(
model_name='profile',
name='location',
field=models.CharField(blank=True, help_text='User location (city, country, etc.)', max_length=200),
),
migrations.AddField(
model_name='profile',
name='pinned',
field=models.CharField(blank=True, help_text='Pinned post ID (timestamp)', max_length=50),
),
]

View file

@ -0,0 +1,31 @@
# Generated by Django 6.0.7 on 2026-07-16 08:13
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('feeds', '0010_profile_birthday_profile_language_profile_location_and_more'),
]
operations = [
migrations.CreateModel(
name='OutgoingWebmention',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('source', models.URLField(help_text='Post URL (feed#post_id) containing the link', max_length=500)),
('target', models.URLField(help_text='External URL the post links to', max_length=500)),
('endpoint', models.URLField(blank=True, help_text='Discovered Webmention endpoint', max_length=500)),
('status', models.CharField(choices=[('pending', 'Pending'), ('sent', 'Sent'), ('failed', 'Failed'), ('no_endpoint', 'No endpoint')], default='pending', max_length=20)),
('attempts', models.PositiveSmallIntegerField(default=0)),
('response_code', models.IntegerField(blank=True, help_text='HTTP status of the last delivery attempt', null=True)),
('last_attempt_at', models.DateTimeField(blank=True, null=True)),
('created_at', models.DateTimeField(auto_now_add=True)),
],
options={
'ordering': ['created_at'],
'unique_together': {('source', 'target')},
},
),
]

View file

@ -0,0 +1,24 @@
from django.db import migrations
from django.db.models import Q
def delete_bridge_feeds(apps, schema_editor):
"""
Bridge virtual feeds are a connection helper, not real accounts.
Remove any that were registered before they were excluded from
registration and discovery.
"""
Feed = apps.get_model("feeds", "Feed")
Feed.objects.filter(
Q(url__contains="/bridge/activitypub/") | Q(url__contains="/bridge/rss/")
).delete()
class Migration(migrations.Migration):
dependencies = [
("feeds", "0011_outgoingwebmention"),
]
operations = [
migrations.RunPython(delete_bridge_feeds, migrations.RunPython.noop),
]

View file

@ -1,4 +1,5 @@
from django.db import models from django.db import models
from django.utils import timezone
import logging import logging
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@ -18,6 +19,20 @@ class Profile(models.Model):
avatar = models.URLField( avatar = models.URLField(
blank=True, help_text="URL to avatar image (128x128px JPG/PNG)" blank=True, help_text="URL to avatar image (128x128px JPG/PNG)"
) )
location = models.CharField(
max_length=200, blank=True, help_text="User location (city, country, etc.)"
)
birthday = models.DateField(
blank=True, null=True, help_text="User birthday in YYYY-MM-DD format"
)
language = models.CharField(
max_length=100,
blank=True,
help_text="Space-separated language codes (ISO 639-1)",
)
pinned = models.CharField(
max_length=50, blank=True, help_text="Pinned post ID (timestamp)"
)
version = models.CharField( version = models.CharField(
max_length=50, max_length=50,
blank=True, blank=True,
@ -133,7 +148,10 @@ class Post(models.Model):
) )
# Metadata # Metadata
created_at = models.DateTimeField(auto_now_add=True) created_at = models.DateTimeField(
default=timezone.now,
help_text="Creation timestamp, parsed from post_id when available",
)
updated_at = models.DateTimeField(auto_now=True) updated_at = models.DateTimeField(auto_now=True)
class Meta: class Meta:
@ -239,6 +257,51 @@ class Feed(models.Model):
return self.url return self.url
class OutgoingWebmention(models.Model):
"""
Outbox of Webmentions sent by the relay on behalf of feed authors.
The (source, target) pair is unique, which guarantees each link found
in a post is notified at most once no matter how many times the feed
is rescanned.
"""
STATUS_PENDING = "pending"
STATUS_SENT = "sent"
STATUS_FAILED = "failed"
STATUS_NO_ENDPOINT = "no_endpoint"
STATUS_CHOICES = [
(STATUS_PENDING, "Pending"),
(STATUS_SENT, "Sent"),
(STATUS_FAILED, "Failed"),
(STATUS_NO_ENDPOINT, "No endpoint"),
]
source = models.URLField(
max_length=500, help_text="Post URL (feed#post_id) containing the link"
)
target = models.URLField(max_length=500, help_text="External URL the post links to")
endpoint = models.URLField(
max_length=500, blank=True, help_text="Discovered Webmention endpoint"
)
status = models.CharField(
max_length=20, choices=STATUS_CHOICES, default=STATUS_PENDING
)
attempts = models.PositiveSmallIntegerField(default=0)
response_code = models.IntegerField(
null=True, blank=True, help_text="HTTP status of the last delivery attempt"
)
last_attempt_at = models.DateTimeField(null=True, blank=True)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
unique_together = ["source", "target"]
ordering = ["created_at"]
def __str__(self):
return f"{self.source} -> {self.target} ({self.status})"
class RelayMetadata(models.Model): class RelayMetadata(models.Model):
""" """
Global metadata for the relay - used for HTTP caching headers. Global metadata for the relay - used for HTTP caching headers.

View file

@ -1,11 +1,38 @@
import re import re
import requests import requests
from datetime import datetime
from typing import Dict, Any, Tuple from typing import Dict, Any, Tuple
from django.utils import timezone from django.utils import timezone
import logging import logging
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
# (connect, read) timeouts in seconds for fetching remote feeds.
# A short connect timeout drops unreachable hosts quickly (e.g. dead servers),
# while the read timeout stays generous enough for slow-but-alive servers.
FEED_FETCH_TIMEOUT = (3.05, 5)
def _sanitize_birthday(value: str) -> str:
"""Return the birthday only if it is a valid YYYY-MM-DD date, else "".
Some feeds use other date formats (e.g. "2003/06/17"). The Profile.birthday
DateField only accepts YYYY-MM-DD, so an invalid value would raise a
ValidationError and abort the whole feed scan. We drop the malformed value
instead of discarding an otherwise valid feed.
"""
value = (value or "").strip()
if not value:
return ""
try:
datetime.strptime(value, "%Y-%m-%d")
except ValueError:
logger.warning(
f"Ignoring invalid birthday value (expected YYYY-MM-DD): {value!r}"
)
return ""
return value
def _update_feed_last_successful_fetch(url: str): def _update_feed_last_successful_fetch(url: str):
""" """
@ -110,11 +137,34 @@ def _handle_feed_redirect(old_url: str, new_url: str):
follow.delete() follow.delete()
logger.debug("Deleted duplicate follow relationship") logger.debug("Deleted duplicate follow relationship")
# Update all Mentions pointing to old_profile # Migrate Mention relationships pointing to old_profile
# Mentions don't have unique constraints, so bulk update is safe # Mentions have unique constraint (post, mentioned_profile)
Mention.objects.filter(mentioned_profile=old_profile).update( mentions_to_migrate = Mention.objects.filter(
mentioned_profile=new_profile mentioned_profile=old_profile
) )
for mention in mentions_to_migrate:
# Check if this mention already exists with new_profile
existing = Mention.objects.filter(
post=mention.post, mentioned_profile=new_profile
).first()
if not existing:
# Update to point to new_profile
mention.mentioned_profile = new_profile
try:
mention.save()
logger.debug(
f"Migrated mention in post {mention.post.post_id}"
)
except Exception as e:
logger.warning(
f"Could not migrate mention, deleting: {e}"
)
mention.delete()
else:
# Mention already exists, delete duplicate
mention.delete()
logger.debug("Deleted duplicate mention")
# Migrate posts from old_profile to new_profile (avoid duplicates) # Migrate posts from old_profile to new_profile (avoid duplicates)
old_posts = Post.objects.filter(profile=old_profile) old_posts = Post.objects.filter(profile=old_profile)
@ -215,7 +265,7 @@ def parse_org_social(url: str) -> Dict[str, Any]:
Dictionary containing parsed metadata and posts Dictionary containing parsed metadata and posts
""" """
try: try:
response = requests.get(url, timeout=5) response = requests.get(url, timeout=FEED_FETCH_TIMEOUT)
response.raise_for_status() response.raise_for_status()
# Decode content as UTF-8 explicitly to avoid encoding issues # Decode content as UTF-8 explicitly to avoid encoding issues
# when the server doesn't specify charset in Content-Type header # when the server doesn't specify charset in Content-Type header
@ -246,6 +296,10 @@ def parse_org_social(url: str) -> Dict[str, Any]:
"nick": "", "nick": "",
"description": "", "description": "",
"avatar": "", "avatar": "",
"location": "",
"birthday": "",
"language": "",
"pinned": "",
"links": [], "links": [],
"follows": [], "follows": [],
"contacts": [], "contacts": [],
@ -274,6 +328,25 @@ def parse_org_social(url: str) -> Dict[str, Any]:
avatar_match = re.search(r"^\s*\#\+AVATAR:\s*(.+)$", content, re.MULTILINE) avatar_match = re.search(r"^\s*\#\+AVATAR:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["avatar"] = avatar_match.group(1).strip() if avatar_match else "" result["metadata"]["avatar"] = avatar_match.group(1).strip() if avatar_match else ""
# Parse new v1.6 fields
location_match = re.search(r"^\s*\#\+LOCATION:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["location"] = (
location_match.group(1).strip() if location_match else ""
)
birthday_match = re.search(r"^\s*\#\+BIRTHDAY:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["birthday"] = (
_sanitize_birthday(birthday_match.group(1)) if birthday_match else ""
)
language_match = re.search(r"^\s*\#\+LANGUAGE:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["language"] = (
language_match.group(1).strip() if language_match else ""
)
pinned_match = re.search(r"^\s*\#\+PINNED:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["pinned"] = pinned_match.group(1).strip() if pinned_match else ""
# Parse multiple values # Parse multiple values
result["metadata"]["links"] = [ result["metadata"]["links"] = [
match.group(1).strip() match.group(1).strip()
@ -305,14 +378,18 @@ def parse_org_social(url: str) -> Dict[str, Any]:
# Split posts by ** headers (exactly 2 asterisks, not 3+) # Split posts by ** headers (exactly 2 asterisks, not 3+)
# Use negative lookahead (?!\*) to ensure we don't match *** or **** # Use negative lookahead (?!\*) to ensure we don't match *** or ****
# Use ^ anchor to match ** only at start of line # Use ^ anchor to match ** only at start of line
post_pattern = r"^\*\*(?!\*)[^\n]*\n(?::PROPERTIES:\s*\n((?::[^:\n]+:[^\n]*\n)*):END:\s*\n)?(.*?)(?=^\*\*(?!\*)|\Z)" # Capture group 1: header content (can contain ID in v1.6)
# Capture group 2: properties text
# Capture group 3: post content
post_pattern = r"^\*\*(?!\*)([^\n]*)\n(?::PROPERTIES:\s*\n((?::[^:\n]+:[^\n]*\n)*):END:\s*\n)?(.*?)(?=^\*\*(?!\*)|\Z)"
post_matches = re.finditer( post_matches = re.finditer(
post_pattern, posts_content, re.DOTALL | re.MULTILINE post_pattern, posts_content, re.DOTALL | re.MULTILINE
) )
for post_match in post_matches: for post_match in post_matches:
properties_text = post_match.group(1) or "" header_text = post_match.group(1).strip() if post_match.group(1) else ""
content_text = post_match.group(2).strip() if post_match.group(2) else "" properties_text = post_match.group(2) or ""
content_text = post_match.group(3).strip() if post_match.group(3) else ""
post: Dict[str, Any] = { post: Dict[str, Any] = {
"id": "", "id": "",
@ -322,6 +399,17 @@ def parse_org_social(url: str) -> Dict[str, Any]:
"poll_options": [], "poll_options": [],
} }
# First check if ID is in header (v1.6 feature)
# Header ID takes priority over property drawer ID
if header_text:
# RFC 3339 format: ####-##-##T##:##:##[+-]####
header_id_match = re.match(
r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{2}:?\d{2}$",
header_text,
)
if header_id_match:
post["id"] = header_text
# Parse properties # Parse properties
if properties_text: if properties_text:
# Use [ \t]* instead of \s* to avoid capturing newlines # Use [ \t]* instead of \s* to avoid capturing newlines
@ -332,7 +420,8 @@ def parse_org_social(url: str) -> Dict[str, Any]:
# Only add non-empty properties # Only add non-empty properties
if prop_value: if prop_value:
post["properties"][prop_name] = prop_value post["properties"][prop_name] = prop_value
if prop_name == "id": # If ID not already set from header, use property ID
if prop_name == "id" and not post["id"]:
post["id"] = prop_value post["id"] = prop_value
# Extract mentions from content # Extract mentions from content
@ -372,6 +461,10 @@ def parse_org_social_content(content: str) -> Dict[str, Any]:
"nick": "", "nick": "",
"description": "", "description": "",
"avatar": "", "avatar": "",
"location": "",
"birthday": "",
"language": "",
"pinned": "",
"links": [], "links": [],
"follows": [], "follows": [],
"contacts": [], "contacts": [],
@ -400,6 +493,25 @@ def parse_org_social_content(content: str) -> Dict[str, Any]:
avatar_match = re.search(r"^\s*\#\+AVATAR:\s*(.+)$", content, re.MULTILINE) avatar_match = re.search(r"^\s*\#\+AVATAR:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["avatar"] = avatar_match.group(1).strip() if avatar_match else "" result["metadata"]["avatar"] = avatar_match.group(1).strip() if avatar_match else ""
# Parse new v1.6 fields
location_match = re.search(r"^\s*\#\+LOCATION:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["location"] = (
location_match.group(1).strip() if location_match else ""
)
birthday_match = re.search(r"^\s*\#\+BIRTHDAY:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["birthday"] = (
_sanitize_birthday(birthday_match.group(1)) if birthday_match else ""
)
language_match = re.search(r"^\s*\#\+LANGUAGE:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["language"] = (
language_match.group(1).strip() if language_match else ""
)
pinned_match = re.search(r"^\s*\#\+PINNED:\s*(.+)$", content, re.MULTILINE)
result["metadata"]["pinned"] = pinned_match.group(1).strip() if pinned_match else ""
# Parse multiple values # Parse multiple values
result["metadata"]["links"] = [ result["metadata"]["links"] = [
match.group(1).strip() match.group(1).strip()
@ -431,14 +543,18 @@ def parse_org_social_content(content: str) -> Dict[str, Any]:
# Split posts by ** headers (exactly 2 asterisks, not 3+) # Split posts by ** headers (exactly 2 asterisks, not 3+)
# Use negative lookahead (?!\*) to ensure we don't match *** or **** # Use negative lookahead (?!\*) to ensure we don't match *** or ****
# Use ^ anchor to match ** only at start of line # Use ^ anchor to match ** only at start of line
post_pattern = r"^\*\*(?!\*)[^\n]*\n(?::PROPERTIES:\s*\n((?::[^:\n]+:[^\n]*\n)*):END:\s*\n)?(.*?)(?=^\*\*(?!\*)|\Z)" # Capture group 1: header content (can contain ID in v1.6)
# Capture group 2: properties text
# Capture group 3: post content
post_pattern = r"^\*\*(?!\*)([^\n]*)\n(?::PROPERTIES:\s*\n((?::[^:\n]+:[^\n]*\n)*):END:\s*\n)?(.*?)(?=^\*\*(?!\*)|\Z)"
post_matches = re.finditer( post_matches = re.finditer(
post_pattern, posts_content, re.DOTALL | re.MULTILINE post_pattern, posts_content, re.DOTALL | re.MULTILINE
) )
for post_match in post_matches: for post_match in post_matches:
properties_text = post_match.group(1) or "" header_text = post_match.group(1).strip() if post_match.group(1) else ""
content_text = post_match.group(2).strip() if post_match.group(2) else "" properties_text = post_match.group(2) or ""
content_text = post_match.group(3).strip() if post_match.group(3) else ""
post: Dict[str, Any] = { post: Dict[str, Any] = {
"id": "", "id": "",
@ -448,6 +564,17 @@ def parse_org_social_content(content: str) -> Dict[str, Any]:
"poll_options": [], "poll_options": [],
} }
# First check if ID is in header (v1.6 feature)
# Header ID takes priority over property drawer ID
if header_text:
# RFC 3339 format: ####-##-##T##:##:##[+-]####
header_id_match = re.match(
r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{2}:?\d{2}$",
header_text,
)
if header_id_match:
post["id"] = header_text
# Parse properties # Parse properties
if properties_text: if properties_text:
# Use [ \t]* instead of \s* to avoid capturing newlines # Use [ \t]* instead of \s* to avoid capturing newlines
@ -458,7 +585,8 @@ def parse_org_social_content(content: str) -> Dict[str, Any]:
# Only add non-empty properties # Only add non-empty properties
if prop_value: if prop_value:
post["properties"][prop_name] = prop_value post["properties"][prop_name] = prop_value
if prop_name == "id": # If ID not already set from header, use property ID
if prop_name == "id" and not post["id"]:
post["id"] = prop_value post["id"] = prop_value
# Extract mentions from content # Extract mentions from content
@ -493,7 +621,7 @@ def validate_org_social_feed(url: str) -> Tuple[bool, str]:
""" """
try: try:
# Check if URL responds with 200 # Check if URL responds with 200
response = requests.get(url, timeout=5) response = requests.get(url, timeout=FEED_FETCH_TIMEOUT)
if response.status_code != 200: if response.status_code != 200:
return False, f"URL returned status code {response.status_code}" return False, f"URL returned status code {response.status_code}"

View file

@ -15,81 +15,36 @@ def discover_feeds_from_relay_nodes():
Periodic task to discover new feeds from other Org Social Relay nodes. Periodic task to discover new feeds from other Org Social Relay nodes.
This task: This task:
1. Fetches the list of relay nodes from the public URL 1. Reads the list of relay nodes from a local file
2. Filters out our own domain to avoid self-discovery 2. Filters out our own domain to avoid self-discovery
3. Calls each relay node's /feeds endpoint to get their registered feeds 3. Calls each relay node's /feeds endpoint to get their registered feeds
4. Stores newly discovered feeds in our local database 4. Stores newly discovered feeds in our local database
""" """
import django import django
from pathlib import Path
django.setup() django.setup()
from django.conf import settings from django.conf import settings
from app.bridge.models import is_bridge_feed_url
from .models import Feed from .models import Feed
from .parser import validate_org_social_feed from .parser import validate_org_social_feed
# URLs to fetch feeds from # Get the project root directory (where manage.py is located)
feed_sources = [ project_root = Path(__file__).resolve().parent.parent.parent
{ relay_list_path = project_root / "relay-list.txt"
"name": "relay nodes",
"url": "https://cdn.jsdelivr.net/gh/tanrax/org-social/org-social-relay-list.txt",
"type": "relay_nodes",
},
{
"name": "public register",
"url": "https://raw.githubusercontent.com/tanrax/org-social/main/registers.txt",
"type": "direct_feeds",
},
]
total_discovered = 0 total_discovered = 0
for source in feed_sources: logger.info(f"Reading relay nodes from: {relay_list_path}")
logger.info(f"Fetching feeds from {source['name']}: {source['url']}")
try: try:
# Fetch the list # Read the local file
response = requests.get(source["url"], timeout=5) with open(relay_list_path, "r", encoding="utf-8") as f:
response.raise_for_status() content = f.read()
# The file might be empty or contain one URL per line # The file might be empty or contain one URL per line
urls = [line.strip() for line in response.text.split("\n") if line.strip()] relay_nodes = [line.strip() for line in content.split("\n") if line.strip()]
if source["type"] == "direct_feeds":
# For direct feeds (registers.txt), validate and add them directly
logger.info(f"Found {len(urls)} direct feeds to validate")
for feed_url in urls:
if not feed_url.strip():
continue
feed_url = feed_url.strip()
# Check if we already have this feed
if Feed.objects.filter(url=feed_url).exists():
continue
# Validate the feed before adding it
logger.info(f"Validating direct feed: {feed_url}")
is_valid, error_message = validate_org_social_feed(feed_url)
if not is_valid:
logger.warning(
f"Skipping invalid direct feed {feed_url}: {error_message}"
)
continue
# Create the feed
try:
Feed.objects.create(url=feed_url)
total_discovered += 1
logger.info(f"Added direct feed: {feed_url}")
except Exception as e:
logger.error(f"Failed to create direct feed {feed_url}: {e}")
elif source["type"] == "relay_nodes":
# For relay nodes, get their feeds endpoints
relay_nodes = urls
# Filter out our own domain to avoid self-discovery # Filter out our own domain to avoid self-discovery
site_domain = settings.SITE_DOMAIN site_domain = settings.SITE_DOMAIN
@ -97,9 +52,7 @@ def discover_feeds_from_relay_nodes():
for node_url in relay_nodes: for node_url in relay_nodes:
# Normalize the URL for comparison # Normalize the URL for comparison
normalized_node = ( normalized_node = (
node_url.replace("http://", "") node_url.replace("http://", "").replace("https://", "").strip("/")
.replace("https://", "")
.strip("/")
) )
normalized_site = site_domain.strip("/") normalized_site = site_domain.strip("/")
@ -111,10 +64,8 @@ def discover_feeds_from_relay_nodes():
relay_nodes = filtered_nodes relay_nodes = filtered_nodes
if not relay_nodes: if not relay_nodes:
logger.info( logger.info("No relay nodes found in the list after filtering own domain")
"No relay nodes found in the list after filtering own domain" return
)
continue
logger.info( logger.info(
f"Found {len(relay_nodes)} relay nodes to check (excluding own domain)" f"Found {len(relay_nodes)} relay nodes to check (excluding own domain)"
@ -141,17 +92,17 @@ def discover_feeds_from_relay_nodes():
if isinstance(feed_url, str) and feed_url.strip(): if isinstance(feed_url, str) and feed_url.strip():
feed_url = feed_url.strip() feed_url = feed_url.strip()
# Bridge virtual feeds are not real accounts
if is_bridge_feed_url(feed_url):
continue
# Check if we already have this feed # Check if we already have this feed
if Feed.objects.filter(url=feed_url).exists(): if Feed.objects.filter(url=feed_url).exists():
continue continue
# Validate the feed before adding it # Validate the feed before adding it
logger.info( logger.info(f"Validating discovered feed: {feed_url}")
f"Validating discovered feed: {feed_url}" is_valid, error_message = validate_org_social_feed(feed_url)
)
is_valid, error_message = validate_org_social_feed(
feed_url
)
if not is_valid: if not is_valid:
logger.warning( logger.warning(
@ -167,29 +118,23 @@ def discover_feeds_from_relay_nodes():
f"Discovered and validated new feed: {feed_url}" f"Discovered and validated new feed: {feed_url}"
) )
except Exception as e: except Exception as e:
logger.error( logger.error(f"Failed to create feed {feed_url}: {e}")
f"Failed to create feed {feed_url}: {e}"
)
logger.info(f"Successfully checked relay node: {node_url}") logger.info(f"Successfully checked relay node: {node_url}")
except requests.RequestException as e: except requests.RequestException as e:
logger.warning( logger.warning(f"Failed to fetch feeds from relay node {node_url}: {e}")
f"Failed to fetch feeds from relay node {node_url}: {e}"
)
except ValueError as e: except ValueError as e:
logger.warning( logger.warning(f"Invalid JSON response from relay node {node_url}: {e}")
f"Invalid JSON response from relay node {node_url}: {e}"
)
except Exception as e: except Exception as e:
logger.error( logger.error(f"Unexpected error checking relay node {node_url}: {e}")
f"Unexpected error checking relay node {node_url}: {e}"
)
except requests.RequestException as e: except FileNotFoundError as e:
logger.error(f"Failed to fetch {source['name']} from {source['url']}: {e}") logger.error(f"Relay list file not found: {relay_list_path}: {e}")
except IOError as e:
logger.error(f"Failed to read relay list from {relay_list_path}: {e}")
except Exception as e: except Exception as e:
logger.error(f"Unexpected error processing {source['name']}: {e}") logger.error(f"Unexpected error processing relay list: {e}")
logger.info( logger.info(
f"Feed discovery completed. Total new feeds discovered: {total_discovered}" f"Feed discovery completed. Total new feeds discovered: {total_discovered}"
@ -211,6 +156,7 @@ def discover_new_feeds_from_follows():
django.setup() django.setup()
from app.bridge.models import is_bridge_feed_url
from .models import Feed, Profile, Follow from .models import Feed, Profile, Follow
from .parser import parse_org_social, validate_org_social_feed from .parser import parse_org_social, validate_org_social_feed
@ -261,6 +207,10 @@ def discover_new_feeds_from_follows():
"nick": metadata.get("nick", ""), "nick": metadata.get("nick", ""),
"description": metadata.get("description", ""), "description": metadata.get("description", ""),
"avatar": metadata.get("avatar", ""), "avatar": metadata.get("avatar", ""),
"location": metadata.get("location", ""),
"birthday": metadata.get("birthday") or None,
"language": metadata.get("language", ""),
"pinned": metadata.get("pinned", ""),
"version": content_hash, "version": content_hash,
}, },
) )
@ -279,7 +229,10 @@ def discover_new_feeds_from_follows():
# Check if feed already exists # Check if feed already exists
existing_feed = Feed.objects.filter(url=follow_url).first() existing_feed = Feed.objects.filter(url=follow_url).first()
if not existing_feed: # Bridge virtual feeds help users connect to external
# content; the follow relationship is kept below, but the
# bridge is never registered as a real feed
if not existing_feed and not is_bridge_feed_url(follow_url):
# Validate the feed before adding it # Validate the feed before adding it
logger.info(f"Validating discovered follow feed: {follow_url}") logger.info(f"Validating discovered follow feed: {follow_url}")
is_valid, error_message = validate_org_social_feed(follow_url) is_valid, error_message = validate_org_social_feed(follow_url)
@ -428,6 +381,10 @@ def scan_feeds():
"nick": metadata.get("nick", ""), "nick": metadata.get("nick", ""),
"description": metadata.get("description", ""), "description": metadata.get("description", ""),
"avatar": metadata.get("avatar", ""), "avatar": metadata.get("avatar", ""),
"location": metadata.get("location", ""),
"birthday": metadata.get("birthday") or None,
"language": metadata.get("language", ""),
"pinned": metadata.get("pinned", ""),
"version": content_hash, "version": content_hash,
}, },
) )
@ -443,13 +400,14 @@ def scan_feeds():
profile.nick = metadata.get("nick", "") profile.nick = metadata.get("nick", "")
profile.description = metadata.get("description", "") profile.description = metadata.get("description", "")
profile.avatar = metadata.get("avatar", "") profile.avatar = metadata.get("avatar", "")
profile.location = metadata.get("location", "")
profile.birthday = metadata.get("birthday") or None
profile.language = metadata.get("language", "")
profile.pinned = metadata.get("pinned", "")
profile.version = content_hash profile.version = content_hash
profile.save() profile.save()
profiles_updated += 1 profiles_updated += 1
logger.info(f"Updated profile: {profile.nick} ({feed.url})") logger.info(f"Updated profile: {profile.nick} ({feed.url})")
else:
# No changes detected, skip processing
continue
# Update profile relationships (clear and recreate) # Update profile relationships (clear and recreate)
profile.links.all().delete() profile.links.all().delete()
@ -482,6 +440,16 @@ def scan_feeds():
content = post_data.get("content", "") content = post_data.get("content", "")
properties = post_data.get("properties", {}) properties = post_data.get("properties", {})
# Parse post_id as timestamp for created_at
# post_id is in RFC 3339 format (e.g., "2025-01-01T12:00:00+00:00")
post_created_at = timezone.now() # Default fallback
try:
post_created_at = date_parser.parse(post_id)
except Exception as e:
logger.warning(
f"Failed to parse post_id {post_id} as timestamp: {e}. Using current time."
)
# Extract group name from GROUP property # Extract group name from GROUP property
# Format: "Emacs https://org-social-relay.andros.dev" or just "Emacs" # Format: "Emacs https://org-social-relay.andros.dev" or just "Emacs"
# Group names can have spaces and capitals - we slugify them # Group names can have spaces and capitals - we slugify them
@ -518,12 +486,23 @@ def scan_feeds():
"group": group_slug, "group": group_slug,
"include": properties.get("include", ""), "include": properties.get("include", ""),
"poll_end": None, "poll_end": None,
"created_at": post_created_at,
}, },
) )
if post_created: if post_created:
posts_created += 1 posts_created += 1
# Queue outgoing webmentions for external links in the post
from .webmentions import queue_webmentions_for_post
try:
queue_webmentions_for_post(profile, post_id, content)
except Exception as e:
logger.warning(
f"Failed to queue webmentions for post {post_id}: {e}"
)
# Publish notifications for NEW posts # Publish notifications for NEW posts
from .notification_publisher import publish_notification from .notification_publisher import publish_notification
@ -565,6 +544,7 @@ def scan_feeds():
) )
else: else:
# Update existing post # Update existing post
content_changed = post.content != content
post.content = content post.content = content
post.language = properties.get("lang", "") post.language = properties.get("lang", "")
post.tags = properties.get("tags", "") post.tags = properties.get("tags", "")
@ -576,6 +556,18 @@ def scan_feeds():
post.save() post.save()
posts_updated += 1 posts_updated += 1
# Links added by an edit get their webmention queued too;
# already-known (source, target) pairs are never re-sent
if content_changed:
from .webmentions import queue_webmentions_for_post
try:
queue_webmentions_for_post(profile, post_id, content)
except Exception as e:
logger.warning(
f"Failed to queue webmentions for post {post_id}: {e}"
)
# Handle poll_end if present # Handle poll_end if present
poll_end_str = properties.get("poll_end", "") poll_end_str = properties.get("poll_end", "")
if poll_end_str: if poll_end_str:
@ -688,6 +680,27 @@ def scan_feeds():
f"Failed to create mention for {mention_url} in post {post_id}: {e}" f"Failed to create mention for {mention_url} in post {post_id}: {e}"
) )
# Detect and remove deleted posts
# Get all post IDs from the current feed scan
current_post_ids = {post_data.get("id", "") for post_data in posts_data}
current_post_ids.discard("") # Remove empty IDs
# Get all post IDs currently in database for this profile
existing_posts = Post.objects.filter(profile=profile)
existing_post_ids = set(existing_posts.values_list("post_id", flat=True))
# Find posts that are in DB but not in current feed (deleted posts)
deleted_post_ids = existing_post_ids - current_post_ids
if deleted_post_ids:
# Delete posts that no longer exist in the feed
deleted_count = Post.objects.filter(
profile=profile, post_id__in=deleted_post_ids
).delete()[0]
logger.info(
f"Removed {deleted_count} deleted post(s) from {feed.url}: {deleted_post_ids}"
)
except requests.RequestException as e: except requests.RequestException as e:
failed_scans += 1 failed_scans += 1
logger.warning(f"Failed to fetch/parse feed {feed.url}: {e}") logger.warning(f"Failed to fetch/parse feed {feed.url}: {e}")
@ -726,6 +739,128 @@ def scan_feeds():
logger.info("Cache cleared after feed scanning - next requests will get fresh data") logger.info("Cache cleared after feed scanning - next requests will get fresh data")
WEBMENTION_MAX_ATTEMPTS = 5
WEBMENTION_BATCH_SIZE = 50
def _send_pending_webmentions_impl():
"""
Implementation of the webmention delivery logic, separated from the
periodic task to allow for easier testing.
Returns:
dict: Counters of sent / no_endpoint / failed webmentions
"""
from .models import OutgoingWebmention
from .webmentions import (
discover_webmention_endpoint,
is_safe_endpoint,
send_webmention,
)
now = timezone.now()
candidates = OutgoingWebmention.objects.filter(
status__in=[
OutgoingWebmention.STATUS_PENDING,
OutgoingWebmention.STATUS_FAILED,
],
attempts__lt=WEBMENTION_MAX_ATTEMPTS,
).order_by("created_at")[: WEBMENTION_BATCH_SIZE * 2]
counters = {"sent": 0, "no_endpoint": 0, "failed": 0}
processed = 0
for webmention in candidates:
if processed >= WEBMENTION_BATCH_SIZE:
break
# Exponential backoff between retries: 1h, 2h, 4h, 8h...
if webmention.last_attempt_at:
backoff = timedelta(hours=2 ** (webmention.attempts - 1))
if now < webmention.last_attempt_at + backoff:
continue
processed += 1
webmention.last_attempt_at = now
webmention.attempts += 1
try:
endpoint = webmention.endpoint or discover_webmention_endpoint(
webmention.target
)
if endpoint is None:
# Permanent: the target does not support webmentions
webmention.status = OutgoingWebmention.STATUS_NO_ENDPOINT
counters["no_endpoint"] += 1
webmention.save()
continue
if not is_safe_endpoint(endpoint):
logger.warning(
f"Rejecting unsafe webmention endpoint {endpoint} "
f"for target {webmention.target}"
)
webmention.status = OutgoingWebmention.STATUS_NO_ENDPOINT
counters["no_endpoint"] += 1
webmention.save()
continue
webmention.endpoint = endpoint
status_code = send_webmention(
endpoint, webmention.source, webmention.target
)
webmention.response_code = status_code
if 200 <= status_code < 300:
webmention.status = OutgoingWebmention.STATUS_SENT
counters["sent"] += 1
logger.info(
f"Webmention sent: {webmention.source} -> {webmention.target} "
f"({status_code})"
)
else:
webmention.status = OutgoingWebmention.STATUS_FAILED
counters["failed"] += 1
logger.warning(
f"Webmention rejected by {endpoint} with HTTP {status_code} "
f"({webmention.source} -> {webmention.target})"
)
except requests.RequestException as e:
webmention.status = OutgoingWebmention.STATUS_FAILED
counters["failed"] += 1
logger.warning(f"Webmention delivery error for {webmention.target}: {e}")
webmention.save()
if processed:
logger.info(
f"Webmention delivery completed. "
f"Sent: {counters['sent']}, "
f"No endpoint: {counters['no_endpoint']}, "
f"Failed: {counters['failed']}"
)
return counters
@periodic_task(crontab(minute="*/5")) # Run every 5 minutes
def send_pending_webmentions():
"""
Periodic task to deliver queued outgoing webmentions.
For each pending (or retriable failed) webmention this task discovers
the target's webmention endpoint and POSTs source/target to it, as
described in https://www.w3.org/TR/webmention/. Targets without an
endpoint are marked permanently so they are never fetched again.
"""
import django
django.setup()
return _send_pending_webmentions_impl()
def _cleanup_stale_feeds_impl(): def _cleanup_stale_feeds_impl():
""" """
Implementation of stale feed cleanup logic. Implementation of stale feed cleanup logic.

View file

@ -0,0 +1,231 @@
from django.test import TestCase
from django.conf import settings
from unittest.mock import patch, Mock
from app.feeds.models import Feed
from app.feeds.tasks import discover_feeds_from_relay_nodes
class DiscoverRelayNodesTest(TestCase):
"""Test cases for discover_feeds_from_relay_nodes task using Given/When/Then structure."""
def setUp(self):
"""Set up test fixtures."""
# Clear any existing feeds
Feed.objects.all().delete()
def test_discovers_feeds_from_real_relay_list(self):
"""Test that feeds are discovered from the actual relay list file."""
# Given: The real relay list file exists (data/relay-list.txt)
# And we mock the HTTP responses from relays
# Mock relay response
mock_relay_response = Mock()
mock_relay_response.status_code = 200
mock_relay_response.json.return_value = {
"type": "Success",
"data": [
"https://test-feed1.com/social.org",
"https://test-feed2.com/social.org",
],
}
mock_relay_response.raise_for_status = Mock()
# Mock feed validation responses
mock_feed_response = Mock()
mock_feed_response.status_code = 200
mock_feed_response.text = """#+TITLE: Test Feed
#+NICK: testuser
#+DESCRIPTION: Test description
* Posts
"""
mock_feed_response.content = mock_feed_response.text.encode("utf-8")
mock_feed_response.history = []
def mock_requests_get(url, timeout=None):
if "/feeds" in url:
return mock_relay_response
# Feed validation requests
mock_feed_response.url = url
mock_feed_response.raise_for_status = Mock()
return mock_feed_response
# When: We run the discovery task
with patch("requests.get", side_effect=mock_requests_get):
discover_feeds_from_relay_nodes()
# Then: Feeds should be discovered (the real relay list has relays)
# We can't predict exact count since it depends on real file content
# So we just verify the function runs without errors
self.assertGreaterEqual(Feed.objects.count(), 0)
def test_handles_relay_connection_failure(self):
"""Test that the task handles relay connection failures gracefully."""
# Given: The real relay list file exists
# And relay connections will fail
# Mock a connection error
with patch("requests.get", side_effect=Exception("Connection failed")):
# When: We run the discovery task
initial_count = Feed.objects.count()
discover_feeds_from_relay_nodes()
# Then: No new feeds should be created and no exception raised
self.assertEqual(Feed.objects.count(), initial_count)
def test_skips_existing_feeds(self):
"""Test that the task skips feeds that already exist."""
# Given: A feed already exists
Feed.objects.create(url="https://existing-feed.com/social.org")
initial_count = Feed.objects.count()
# Mock relay response with the existing feed
mock_relay_response = Mock()
mock_relay_response.status_code = 200
mock_relay_response.json.return_value = {
"type": "Success",
"data": [
"https://existing-feed.com/social.org", # Already exists
],
}
mock_relay_response.raise_for_status = Mock()
def mock_requests_get(url, timeout=None):
if "/feeds" in url:
return mock_relay_response
# Should not validate existing feed
if "existing-feed" in url:
self.fail("Should not validate existing feed")
return Mock()
# When: We run the discovery task
with patch("requests.get", side_effect=mock_requests_get):
discover_feeds_from_relay_nodes()
# Then: No new feeds should be added
self.assertEqual(Feed.objects.count(), initial_count)
def test_handles_invalid_feed_during_validation(self):
"""Test that the task skips invalid feeds during validation."""
# Given: The real relay list file exists
# Mock relay response with valid and invalid feeds
mock_relay_response = Mock()
mock_relay_response.status_code = 200
mock_relay_response.json.return_value = {
"type": "Success",
"data": [
"https://valid-feed.com/social.org",
"https://invalid-feed.com/social.org",
],
}
mock_relay_response.raise_for_status = Mock()
# Mock feed validation responses
def mock_requests_get(url, timeout=None):
if "/feeds" in url:
return mock_relay_response
# Valid feed response
if "valid-feed" in url:
mock_response = Mock()
mock_response.status_code = 200
mock_response.text = """#+TITLE: Valid Feed
#+NICK: validuser
#+DESCRIPTION: Valid description
* Posts
"""
mock_response.content = mock_response.text.encode("utf-8")
mock_response.url = url
mock_response.history = []
mock_response.raise_for_status = Mock()
return mock_response
# Invalid feed response (missing required fields)
if "invalid-feed" in url:
mock_response = Mock()
mock_response.status_code = 200
mock_response.text = "Invalid content without Org Social headers"
mock_response.content = mock_response.text.encode("utf-8")
mock_response.url = url
mock_response.history = []
mock_response.raise_for_status = Mock()
return mock_response
# Default response for any other URL
mock_response = Mock()
mock_response.status_code = 404
mock_response.raise_for_status = Mock(side_effect=Exception("404"))
return mock_response
# When: We run the discovery task
initial_count = Feed.objects.count()
with patch("requests.get", side_effect=mock_requests_get):
discover_feeds_from_relay_nodes()
# Then: Only the valid feed should be added (or none if validation failed for both)
# We verify that invalid feed was not added
self.assertFalse(
Feed.objects.filter(url="https://invalid-feed.com/social.org").exists()
)
# And if valid feed was processed, it should exist
# (we're more lenient here as the test DB might interfere)
if Feed.objects.count() > initial_count:
self.assertTrue(
Feed.objects.filter(url="https://valid-feed.com/social.org").exists()
)
def test_filters_out_own_domain(self):
"""Test that the task filters out its own domain from relay list."""
# Given: The real relay list file exists
# We verify the function doesn't try to fetch from its own domain
own_domain = settings.SITE_DOMAIN
def mock_requests_get(url, timeout=None):
# Should not be called for our own domain
if own_domain in url and "/feeds" in url:
self.fail(f"Should not fetch from own domain: {url}")
# Mock response for other relays
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {"type": "Success", "data": []}
mock_response.raise_for_status = Mock()
return mock_response
# When: We run the discovery task
with patch("requests.get", side_effect=mock_requests_get):
discover_feeds_from_relay_nodes()
# Then: The function should complete without calling own domain
# (if it called own domain, the test would fail in mock_requests_get)
def test_skips_bridge_feeds_from_relay_nodes(self):
"""Test that bridge virtual feeds listed by other relays are skipped."""
# Given: A relay node listing only bridge virtual feeds
mock_relay_response = Mock()
mock_relay_response.status_code = 200
mock_relay_response.json.return_value = {
"type": "Success",
"data": [
"https://other-relay.org/bridge/rss/"
"?url=https%3A%2F%2Frss.arxiv.org%2Frss%2Fquant-ph",
"https://other-relay.org/bridge/activitypub/@user@instance.tld/",
],
}
mock_relay_response.raise_for_status = Mock()
def mock_requests_get(url, timeout=None):
if "/feeds" in url:
return mock_relay_response
# Bridge feeds must not even be validated
self.fail(f"Should not validate bridge feed: {url}")
# When: We run the discovery task
with patch("requests.get", side_effect=mock_requests_get):
discover_feeds_from_relay_nodes()
# Then: No bridge feed is registered
self.assertEqual(Feed.objects.count(), 0)

View file

@ -339,3 +339,43 @@ class FeedsViewTest(TestCase):
# Then: Should still have caching headers # Then: Should still have caching headers
self.assertIn("ETag", response) self.assertIn("ETag", response)
self.assertIn("Last-Modified", response) self.assertIn("Last-Modified", response)
def test_get_feeds_excludes_bridge_feeds(self):
"""Test GET /feeds never lists bridge virtual feeds."""
# Given: A real feed and two bridge virtual feeds in the database
Feed.objects.create(url="https://example.com/social.org")
Feed.objects.create(
url="https://relay.org-social.org/bridge/rss/"
"?url=https%3A%2F%2Frss.arxiv.org%2Frss%2Fquant-ph"
)
Feed.objects.create(
url="https://relay.org-social.org/bridge/activitypub/@user@instance.tld/"
)
# When: We request the feeds list
response = self.client.get(self.feeds_url)
# Then: Only the real feed is listed
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["data"], ["https://example.com/social.org"])
def test_post_bridge_feed_is_rejected(self):
"""Test POST /feeds rejects bridge virtual feeds."""
# Given: The URL of a bridge virtual feed
bridge_url = (
"https://relay.org-social.org/bridge/rss/"
"?url=https%3A%2F%2Fxkcd.com%2Frss.xml"
)
# When: We try to register it
response = self.client.post(
self.feeds_url, {"feed": bridge_url}, format="json"
)
# Then: The request is rejected and no feed is created
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(response.data["type"], "Error")
self.assertIn(
"Bridge virtual feeds cannot be registered", response.data["errors"]
)
self.assertFalse(Feed.objects.filter(url=bridge_url).exists())

View file

@ -640,4 +640,331 @@ Great work!
# Then: Verify we're using response.content, not response.text # Then: Verify we're using response.content, not response.text
# This is critical to avoid double-encoding # This is critical to avoid double-encoding
mock_get.assert_called_once_with("https://example.com/social.org", timeout=5) from app.feeds.parser import FEED_FETCH_TIMEOUT
mock_get.assert_called_once_with(
"https://example.com/social.org", timeout=FEED_FETCH_TIMEOUT
)
def test_parse_v16_metadata_fields(self):
"""Test parsing v1.6 metadata fields (LOCATION, BIRTHDAY, LANGUAGE, PINNED)."""
# Given: An org social content with v1.6 metadata fields
content = """#+TITLE: Test User Profile
#+NICK: test_user
#+DESCRIPTION: Test description
#+AVATAR: https://example.com/avatar.jpg
#+LOCATION: Valencia, Spain
#+BIRTHDAY: 1990-05-15
#+LANGUAGE: en es ca
#+PINNED: 2025-01-15T10:00:00+0100
* Posts
** 2025-01-15T10:00:00+0100
:PROPERTIES:
:END:
This is my pinned post.
** 2025-01-16T12:00:00+0100
:PROPERTIES:
:END:
This is a regular post.
"""
# When: We parse the content
from app.feeds.parser import parse_org_social_content
result = parse_org_social_content(content)
# Then: All v1.6 metadata fields should be correctly parsed
self.assertEqual(result["metadata"]["location"], "Valencia, Spain")
self.assertEqual(result["metadata"]["birthday"], "1990-05-15")
self.assertEqual(result["metadata"]["language"], "en es ca")
self.assertEqual(result["metadata"]["pinned"], "2025-01-15T10:00:00+0100")
# Then: Posts should be parsed correctly
self.assertEqual(len(result["posts"]), 2)
def test_parse_invalid_birthday_is_dropped(self):
"""An invalid birthday format must be dropped, not abort the feed."""
# Given: A feed with a birthday that is not in YYYY-MM-DD format
content = """#+TITLE: Test User Profile
#+NICK: test_user
#+BIRTHDAY: 2003/06/17
* Posts
** 2025-01-15T10:00:00+0100
:PROPERTIES:
:END:
A post.
"""
# When: We parse the content
from app.feeds.parser import parse_org_social_content
result = parse_org_social_content(content)
# Then: The malformed birthday is dropped but the feed is still parsed
self.assertEqual(result["metadata"]["birthday"], "")
self.assertEqual(result["metadata"]["nick"], "test_user")
self.assertEqual(len(result["posts"]), 1)
def test_parse_post_id_in_header(self):
"""Test parsing post ID from header (v1.6 feature)."""
# Given: An org social content with post ID in header
content = """#+TITLE: Test
#+NICK: test_user
* Posts
** 2025-01-15T10:00:00+0100
:PROPERTIES:
:LANG: en
:END:
This post has ID in the header.
**
:PROPERTIES:
:ID: 2025-01-16T12:00:00+0100
:END:
This post has ID in properties.
"""
# When: We parse the content
from app.feeds.parser import parse_org_social_content
result = parse_org_social_content(content)
# Then: Both posts should have correct IDs
self.assertEqual(len(result["posts"]), 2)
# Then: First post should have ID from header
post1 = result["posts"][0]
self.assertEqual(post1["id"], "2025-01-15T10:00:00+0100")
# Then: Second post should have ID from properties
post2 = result["posts"][1]
self.assertEqual(post2["id"], "2025-01-16T12:00:00+0100")
def test_parse_post_id_priority_header_over_property(self):
"""Test that header ID takes priority over property ID (v1.6 spec)."""
# Given: An org social content with post ID in both header and properties
content = """#+TITLE: Test
#+NICK: test_user
* Posts
** 2025-01-15T10:00:00+0100
:PROPERTIES:
:ID: 2025-01-16T12:00:00+0100
:END:
This post has ID in both places. Header should take priority.
"""
# When: We parse the content
from app.feeds.parser import parse_org_social_content
result = parse_org_social_content(content)
# Then: Post should have ID from header (priority)
self.assertEqual(len(result["posts"]), 1)
post = result["posts"][0]
self.assertEqual(post["id"], "2025-01-15T10:00:00+0100")
@patch("app.feeds.parser.requests.get")
def test_parse_feed_with_301_redirect(self, mock_get):
"""Test that feed redirects (301) are properly detected and handled."""
# Given: A feed URL that redirects to a new URL
old_url = "https://old.example.com/social.org"
new_url = "https://new.example.com/social.org"
content = """#+TITLE: Test User
#+NICK: test_user
* Posts
** 2025-01-15T10:00:00+0100
:PROPERTIES:
:END:
Test post
"""
# Given: Mock response with redirect history
mock_redirect = Mock()
mock_redirect.status_code = 301
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = content.encode("utf-8")
mock_response.url = new_url # Final URL after redirect
mock_response.history = [mock_redirect] # Redirect happened
mock_response.raise_for_status = Mock()
mock_get.return_value = mock_response
# When: We parse the feed from old URL
from app.feeds.parser import parse_org_social
result = parse_org_social(old_url)
# Then: The content should be parsed correctly
self.assertEqual(result["metadata"]["nick"], "test_user")
self.assertEqual(len(result["posts"]), 1)
@patch("app.feeds.parser.requests.get")
def test_validate_feed_with_301_redirect(self, mock_get):
"""Test that feed validation handles redirects properly."""
# Given: A feed URL that redirects to a new URL
old_url = "https://old.example.com/social.org"
new_url = "https://new.example.com/social.org"
content = """#+TITLE: Test User
#+NICK: test_user
* Posts
"""
# Given: Mock response with redirect history
mock_redirect = Mock()
mock_redirect.status_code = 301
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = content.encode("utf-8")
mock_response.url = new_url # Final URL after redirect
mock_response.history = [mock_redirect] # Redirect happened
mock_response.raise_for_status = Mock()
mock_get.return_value = mock_response
# When: We validate the feed
from app.feeds.parser import validate_org_social_feed
is_valid, error_message = validate_org_social_feed(old_url)
# Then: The feed should be valid
self.assertTrue(is_valid)
self.assertEqual(error_message, "")
class FeedRedirectHandlerTest(TestCase):
"""Test cases for feed redirect handling logic."""
def test_handle_redirect_updates_feed_url_when_only_old_exists(self):
"""Test updating feed URL when only old feed exists."""
# Given: A feed exists with old URL
from app.feeds.models import Feed, Profile
from app.feeds.parser import _handle_feed_redirect
old_url = "https://old.example.com/social.org"
new_url = "https://new.example.com/social.org"
Feed.objects.create(url=old_url)
Profile.objects.create(
feed=old_url, nick="testuser", title="Test User", version="v1"
)
# When: Redirect is handled
_handle_feed_redirect(old_url, new_url)
# Then: Feed URL should be updated
self.assertFalse(Feed.objects.filter(url=old_url).exists())
self.assertTrue(Feed.objects.filter(url=new_url).exists())
# Then: Profile should point to new URL
profile = Profile.objects.get(feed=new_url)
self.assertEqual(profile.nick, "testuser")
def test_handle_redirect_merges_feeds_when_both_exist(self):
"""Test merging feeds when both old and new URLs exist."""
# Given: Both old and new feeds exist
from app.feeds.models import Feed, Profile, Post
from app.feeds.parser import _handle_feed_redirect
from django.utils import timezone
old_url = "https://old.example.com/social.org"
new_url = "https://new.example.com/social.org"
# Create old feed and profile
Feed.objects.create(url=old_url)
old_profile = Profile.objects.create(
feed=old_url, nick="olduser", title="Old User", version="v1"
)
Post.objects.create(
profile=old_profile,
post_id="2025-01-15T10:00:00+0100",
content="Old post",
created_at=timezone.now(),
)
# Create new feed and profile
Feed.objects.create(url=new_url)
new_profile = Profile.objects.create(
feed=new_url, nick="newuser", title="New User", version="v2"
)
# When: Redirect is handled
_handle_feed_redirect(old_url, new_url)
# Then: Old feed should be deleted
self.assertFalse(Feed.objects.filter(url=old_url).exists())
self.assertTrue(Feed.objects.filter(url=new_url).exists())
# Then: Old profile should be deleted
self.assertFalse(Profile.objects.filter(feed=old_url).exists())
# Then: Post should be migrated to new profile
post = Post.objects.get(post_id="2025-01-15T10:00:00+0100")
self.assertEqual(post.profile, new_profile)
def test_handle_redirect_merges_mentions_without_duplicates(self):
"""Test that mentions are merged correctly, avoiding UNIQUE constraint errors."""
# Given: Both old and new profiles exist with mentions
from app.feeds.models import Feed, Profile, Post, Mention
from app.feeds.parser import _handle_feed_redirect
from django.utils import timezone
old_url = "https://old.example.com/social.org"
new_url = "https://new.example.com/social.org"
# Create old feed and profile
Feed.objects.create(url=old_url)
old_profile = Profile.objects.create(
feed=old_url, nick="olduser", title="Old User", version="v1"
)
# Create new feed and profile
Feed.objects.create(url=new_url)
new_profile = Profile.objects.create(
feed=new_url, nick="newuser", title="New User", version="v2"
)
# Create a third profile that mentions both old and new profiles
third_profile = Profile.objects.create(
feed="https://third.example.com/social.org",
nick="third",
title="Third User",
version="v1",
)
post = Post.objects.create(
profile=third_profile,
post_id="2025-01-15T10:00:00+0100",
content="Mentioning both profiles",
created_at=timezone.now(),
)
# Create mentions: one to old_profile, one to new_profile
Mention.objects.create(post=post, mentioned_profile=old_profile, nickname="old")
Mention.objects.create(post=post, mentioned_profile=new_profile, nickname="new")
# When: Redirect is handled (this should NOT raise UNIQUE constraint error)
_handle_feed_redirect(old_url, new_url)
# Then: Old profile should be deleted
self.assertFalse(Profile.objects.filter(feed=old_url).exists())
# Then: Only one mention should remain (the duplicate was deleted)
mentions = Mention.objects.filter(post=post)
self.assertEqual(mentions.count(), 1)
self.assertEqual(mentions.first().mentioned_profile, new_profile)

View file

@ -0,0 +1,217 @@
from unittest.mock import Mock, patch
from django.test import TestCase
from app.feeds.models import Feed, Profile, Post
from app.feeds.tasks import scan_feeds
class PostDeletionDetectionTest(TestCase):
"""Test cases for detecting and removing deleted posts."""
def setUp(self):
"""Set up test fixtures."""
self.feed_url = "https://example.com/social.org"
self.feed = Feed.objects.create(url=self.feed_url)
self.profile = Profile.objects.create(
feed=self.feed_url,
nick="test_user",
title="Test User",
)
def test_detect_deleted_posts(self):
"""Test deletion detection logic."""
# Given: Profile has 3 posts in database
Post.objects.create(
profile=self.profile, post_id="2025-01-01T10:00:00+0100", content="Post 1"
)
Post.objects.create(
profile=self.profile, post_id="2025-01-02T10:00:00+0100", content="Post 2"
)
Post.objects.create(
profile=self.profile, post_id="2025-01-03T10:00:00+0100", content="Post 3"
)
# When: Current feed only has 2 posts (simulating one was deleted)
current_post_ids = {"2025-01-01T10:00:00+0100", "2025-01-03T10:00:00+0100"}
# Get existing posts from database
existing_posts = Post.objects.filter(profile=self.profile)
existing_post_ids = set(existing_posts.values_list("post_id", flat=True))
# Find deleted posts
deleted_post_ids = existing_post_ids - current_post_ids
# Then: Should detect one deleted post
self.assertEqual(len(deleted_post_ids), 1)
self.assertIn("2025-01-02T10:00:00+0100", deleted_post_ids)
# When: Delete the posts that no longer exist
deleted_count = Post.objects.filter(
profile=self.profile, post_id__in=deleted_post_ids
).delete()[0]
# Then: One post should be deleted
self.assertEqual(deleted_count, 1)
# Then: Only 2 posts should remain
remaining_posts = Post.objects.filter(profile=self.profile)
self.assertEqual(remaining_posts.count(), 2)
# Then: Deleted post should not exist
self.assertFalse(
Post.objects.filter(
profile=self.profile, post_id="2025-01-02T10:00:00+0100"
).exists()
)
# Then: Other posts should still exist
self.assertTrue(
Post.objects.filter(
profile=self.profile, post_id="2025-01-01T10:00:00+0100"
).exists()
)
self.assertTrue(
Post.objects.filter(
profile=self.profile, post_id="2025-01-03T10:00:00+0100"
).exists()
)
def test_no_posts_deleted_when_all_present(self):
"""Test that no posts are deleted when all posts are still in feed."""
# Given: Profile has 2 posts
Post.objects.create(
profile=self.profile, post_id="2025-01-01T10:00:00+0100", content="Post 1"
)
Post.objects.create(
profile=self.profile, post_id="2025-01-02T10:00:00+0100", content="Post 2"
)
# When: Current feed still has both posts
current_post_ids = {"2025-01-01T10:00:00+0100", "2025-01-02T10:00:00+0100"}
# Get existing posts
existing_posts = Post.objects.filter(profile=self.profile)
existing_post_ids = set(existing_posts.values_list("post_id", flat=True))
# Find deleted posts
deleted_post_ids = existing_post_ids - current_post_ids
# Then: No posts should be detected as deleted
self.assertEqual(len(deleted_post_ids), 0)
# Then: Both posts should still exist
self.assertEqual(Post.objects.filter(profile=self.profile).count(), 2)
def test_all_posts_deleted(self):
"""Test handling when all posts are deleted from feed."""
# Given: Profile has 2 posts
Post.objects.create(
profile=self.profile, post_id="2025-01-01T10:00:00+0100", content="Post 1"
)
Post.objects.create(
profile=self.profile, post_id="2025-01-02T10:00:00+0100", content="Post 2"
)
# When: Current feed has no posts
current_post_ids = set()
# Get existing posts
existing_posts = Post.objects.filter(profile=self.profile)
existing_post_ids = set(existing_posts.values_list("post_id", flat=True))
# Find deleted posts
deleted_post_ids = existing_post_ids - current_post_ids
# Then: All posts should be detected as deleted
self.assertEqual(len(deleted_post_ids), 2)
# When: Delete all posts
deleted_count = Post.objects.filter(
profile=self.profile, post_id__in=deleted_post_ids
).delete()[0]
# Then: 2 posts should be deleted
self.assertEqual(deleted_count, 2)
# Then: No posts should remain
self.assertEqual(Post.objects.filter(profile=self.profile).count(), 0)
def test_multiple_posts_deleted(self):
"""Test handling when multiple posts are deleted."""
# Given: Profile has 5 posts
for i in range(1, 6):
Post.objects.create(
profile=self.profile,
post_id=f"2025-01-0{i}T10:00:00+0100",
content=f"Post {i}",
)
# When: Current feed only has 2 posts (3 were deleted)
current_post_ids = {"2025-01-01T10:00:00+0100", "2025-01-05T10:00:00+0100"}
# Get existing posts
existing_posts = Post.objects.filter(profile=self.profile)
existing_post_ids = set(existing_posts.values_list("post_id", flat=True))
# Find deleted posts
deleted_post_ids = existing_post_ids - current_post_ids
# Then: 3 posts should be detected as deleted
self.assertEqual(len(deleted_post_ids), 3)
# When: Delete them
deleted_count = Post.objects.filter(
profile=self.profile, post_id__in=deleted_post_ids
).delete()[0]
# Then: 3 posts should be deleted
self.assertEqual(deleted_count, 3)
# Then: Only 2 posts should remain
self.assertEqual(Post.objects.filter(profile=self.profile).count(), 2)
class ScanFeedsRobustnessTest(TestCase):
"""End-to-end robustness tests for the scan_feeds task."""
@patch("app.feeds.parser.requests.get")
def test_invalid_birthday_does_not_abort_scan(self, mock_get):
"""A feed with a malformed birthday must still be scanned (regression).
Previously a value like "2003/06/17" reached the Profile.birthday
DateField and raised a ValidationError, aborting the whole feed scan on
every run. The birthday must now be dropped while posts are still saved.
"""
# Given: A feed whose birthday is not in YYYY-MM-DD format
feed_url = "https://host.example.org/ali/social.org"
Feed.objects.create(url=feed_url)
content = (
"#+TITLE: Ali\n"
"#+NICK: ali\n"
"#+BIRTHDAY: 2003/06/17\n"
"\n"
"* Posts\n"
"** 2025-01-01T10:00:00+0100\n"
":PROPERTIES:\n"
":END:\n"
"\n"
"Hello world\n"
)
mock_response = Mock()
mock_response.status_code = 200
mock_response.content = content.encode("utf-8")
mock_response.url = feed_url # No redirect
mock_response.history = []
mock_response.raise_for_status = Mock()
mock_get.return_value = mock_response
# When: We scan all feeds
scan_feeds.call_local()
# Then: The profile is created with the bad birthday dropped, posts saved
profile = Profile.objects.get(feed=feed_url)
self.assertEqual(profile.nick, "ali")
self.assertIsNone(profile.birthday)
self.assertEqual(Post.objects.filter(profile=profile).count(), 1)

View file

@ -0,0 +1,656 @@
from datetime import timedelta
from unittest.mock import Mock, patch
import requests
from django.test import TestCase
from django.utils import timezone
from app.feeds.models import Feed, OutgoingWebmention, Profile
from app.feeds.tasks import WEBMENTION_MAX_ATTEMPTS, _send_pending_webmentions_impl
from app.feeds.webmentions import (
discover_webmention_endpoint,
extract_external_urls,
is_safe_endpoint,
queue_webmentions_for_post,
)
class ExtractExternalUrlsTest(TestCase):
"""Test cases for URL extraction from post content."""
def test_extract_org_link_with_description(self):
# Given: A post containing an Org link with description
content = "Great read: [[https://example.com/article][My article]]"
# When: URLs are extracted
result = extract_external_urls(content)
# Then: The linked URL is found without the description
self.assertEqual(result, ["https://example.com/article"])
def test_extract_org_link_without_description(self):
# Given: A post containing a plain Org link
content = "See [[https://example.com/article]] for details"
# When: URLs are extracted
result = extract_external_urls(content)
# Then: The linked URL is found
self.assertEqual(result, ["https://example.com/article"])
def test_extract_bare_url(self):
# Given: A post containing a bare URL
content = "Check https://example.com/post and tell me"
# When: URLs are extracted
result = extract_external_urls(content)
# Then: The bare URL is found
self.assertEqual(result, ["https://example.com/post"])
def test_bare_url_trailing_punctuation_stripped(self):
# Given: A bare URL wrapped in punctuation
content = "Nice one (https://example.com/post). Really!"
# When: URLs are extracted
result = extract_external_urls(content)
# Then: Trailing punctuation is not part of the URL
self.assertEqual(result, ["https://example.com/post"])
def test_url_with_query_string_and_fragment_is_preserved(self):
# Given: A URL with query string and fragment
content = "See https://example.com/post?id=1&lang=en#section-2"
# When: URLs are extracted
result = extract_external_urls(content)
# Then: Query string and fragment are kept intact
self.assertEqual(result, ["https://example.com/post?id=1&lang=en#section-2"])
def test_org_social_mentions_are_ignored(self):
# Given: A post with an Org Social mention and a normal link
content = (
"Hi [[org-social:https://other.example/social.org][bob]] look at "
"[[https://example.com/article][this]]"
)
# When: URLs are extracted
result = extract_external_urls(content)
# Then: Only the normal link is found, the mention is skipped
self.assertEqual(result, ["https://example.com/article"])
def test_duplicate_urls_are_deduplicated(self):
# Given: A post repeating the same URL as Org link and bare URL
content = (
"https://example.com/a and again [[https://example.com/a][same]] "
"plus https://example.com/b"
)
# When: URLs are extracted
result = extract_external_urls(content)
# Then: Each URL appears once, in order of appearance
self.assertEqual(result, ["https://example.com/a", "https://example.com/b"])
def test_empty_or_missing_content_returns_no_urls(self):
# Given: Empty and missing content
# When: URLs are extracted
# Then: No URLs are found and nothing breaks
self.assertEqual(extract_external_urls(""), [])
self.assertEqual(extract_external_urls(None), [])
def test_non_http_schemes_are_ignored(self):
# Given: A post with non-http(s) URIs only
content = "Write to mailto:me@example.com or ftp://example.com/file"
# When: URLs are extracted
result = extract_external_urls(content)
# Then: Nothing is extracted
self.assertEqual(result, [])
def _mock_response(
url="https://example.com/article",
headers=None,
body=b"",
encoding="utf-8",
):
"""Build a minimal mock of a streamed requests.Response."""
response = Mock()
response.url = url
response.headers = headers or {}
response.encoding = encoding
response.raw = Mock()
response.raw.read = Mock(return_value=body)
return response
class DiscoverWebmentionEndpointTest(TestCase):
"""Test cases for endpoint discovery per the W3C spec."""
@patch("app.feeds.webmentions.requests.get")
def test_link_header_discovery(self, mock_get):
# Given: A target advertising its endpoint in the HTTP Link header
mock_get.return_value = _mock_response(
headers={
"Link": '<https://example.com/wm>; rel="webmention"',
"Content-Type": "text/html",
}
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: The Link header endpoint is returned
self.assertEqual(endpoint, "https://example.com/wm")
@patch("app.feeds.webmentions.requests.get")
def test_link_header_takes_precedence_over_html(self, mock_get):
# Given: A target with endpoints both in the Link header and the HTML
mock_get.return_value = _mock_response(
headers={
"Link": '<https://example.com/header-wm>; rel="webmention"',
"Content-Type": "text/html",
},
body=b'<link rel="webmention" href="https://example.com/html-wm">',
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: The Link header wins, as the spec requires
self.assertEqual(endpoint, "https://example.com/header-wm")
@patch("app.feeds.webmentions.requests.get")
def test_link_header_with_multiple_rels(self, mock_get):
# Given: A Link header whose rel lists several values
mock_get.return_value = _mock_response(
headers={
"Link": '<https://example.com/wm>; rel="webmention somethingelse"',
"Content-Type": "text/html",
}
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: The webmention rel is still recognized
self.assertEqual(endpoint, "https://example.com/wm")
@patch("app.feeds.webmentions.requests.get")
def test_html_link_element_discovery(self, mock_get):
# Given: A target advertising a relative endpoint in a <link> element
mock_get.return_value = _mock_response(
headers={"Content-Type": "text/html; charset=utf-8"},
body=b'<html><head><link rel="webmention" href="/wm"></head></html>',
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: The relative href is resolved against the target
self.assertEqual(endpoint, "https://example.com/wm")
@patch("app.feeds.webmentions.requests.get")
def test_html_anchor_element_discovery(self, mock_get):
# Given: A target advertising its endpoint in an <a> element
mock_get.return_value = _mock_response(
headers={"Content-Type": "text/html"},
body=b'<body><a rel="webmention" href="https://wm.example.org/ep">wm</a></body>',
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: The anchor endpoint is returned
self.assertEqual(endpoint, "https://wm.example.org/ep")
@patch("app.feeds.webmentions.requests.get")
def test_first_element_in_document_order_wins(self, mock_get):
# Given: Both an <a> and a <link> endpoint, the <a> appearing first
mock_get.return_value = _mock_response(
headers={"Content-Type": "text/html"},
body=(
b'<a rel="webmention" href="/first">a</a>'
b'<link rel="webmention" href="/second">'
),
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: The first element in document order wins
self.assertEqual(endpoint, "https://example.com/first")
@patch("app.feeds.webmentions.requests.get")
def test_empty_href_resolves_to_page_url(self, mock_get):
# Given: An endpoint advertised with an empty href
mock_get.return_value = _mock_response(
url="https://example.com/article",
headers={"Content-Type": "text/html"},
body=b'<link rel="webmention" href="">',
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: It resolves to the page URL itself
self.assertEqual(endpoint, "https://example.com/article")
@patch("app.feeds.webmentions.requests.get")
def test_relative_endpoint_resolved_after_redirect(self, mock_get):
# Given: A target that redirected and advertises a relative endpoint
mock_get.return_value = _mock_response(
url="https://final.example.com/page",
headers={"Content-Type": "text/html"},
body=b'<link rel="webmention" href="wm-endpoint">',
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: The endpoint resolves against the final URL after redirects
self.assertEqual(endpoint, "https://final.example.com/wm-endpoint")
@patch("app.feeds.webmentions.requests.get")
def test_no_endpoint_returns_none(self, mock_get):
# Given: A target without any webmention endpoint
mock_get.return_value = _mock_response(
headers={"Content-Type": "text/html"},
body=b"<html><body>No webmention here</body></html>",
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: No endpoint is found
self.assertIsNone(endpoint)
@patch("app.feeds.webmentions.requests.get")
def test_non_html_content_without_link_header_returns_none(self, mock_get):
# Given: A non-HTML target without a Link header
mock_get.return_value = _mock_response(
headers={"Content-Type": "application/pdf"},
body=b"%PDF-1.4",
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/doc.pdf")
# Then: No endpoint is found and the body is never parsed as HTML
self.assertIsNone(endpoint)
@patch("app.feeds.webmentions.requests.get")
def test_rel_without_webmention_is_ignored(self, mock_get):
# Given: A page whose only <link> has an unrelated rel
mock_get.return_value = _mock_response(
headers={"Content-Type": "text/html"},
body=b'<link rel="stylesheet" href="/style.css">',
)
# When: The endpoint is discovered
endpoint = discover_webmention_endpoint("https://example.com/article")
# Then: No endpoint is found
self.assertIsNone(endpoint)
class IsSafeEndpointTest(TestCase):
"""Test cases for endpoint safety checks (spec section 4.3)."""
def test_rejects_non_http_scheme(self):
# Given: Endpoints with non-http(s) schemes
# When: They are checked
# Then: They are rejected
self.assertFalse(is_safe_endpoint("ftp://example.com/wm"))
self.assertFalse(is_safe_endpoint("file:///etc/passwd"))
@patch("app.feeds.webmentions.socket.getaddrinfo")
def test_rejects_loopback(self, mock_getaddrinfo):
# Given: An endpoint resolving to a loopback address
mock_getaddrinfo.return_value = [(2, 1, 6, "", ("127.0.0.1", 80))]
# When/Then: It is rejected
self.assertFalse(is_safe_endpoint("http://localhost/wm"))
@patch("app.feeds.webmentions.socket.getaddrinfo")
def test_rejects_private_address(self, mock_getaddrinfo):
# Given: An endpoint resolving to a private address
mock_getaddrinfo.return_value = [(2, 1, 6, "", ("192.168.1.10", 80))]
# When/Then: It is rejected
self.assertFalse(is_safe_endpoint("https://internal.example.com/wm"))
@patch("app.feeds.webmentions.socket.getaddrinfo")
def test_accepts_public_address(self, mock_getaddrinfo):
# Given: An endpoint resolving to a public address
mock_getaddrinfo.return_value = [(2, 1, 6, "", ("93.184.216.34", 443))]
# When/Then: It is accepted
self.assertTrue(is_safe_endpoint("https://example.com/wm"))
@patch("app.feeds.webmentions.socket.getaddrinfo")
def test_rejects_unresolvable_host(self, mock_getaddrinfo):
# Given: An endpoint whose host does not resolve
import socket as socket_module
mock_getaddrinfo.side_effect = socket_module.gaierror()
# When/Then: It is rejected
self.assertFalse(is_safe_endpoint("https://nope.invalid/wm"))
class QueueWebmentionsForPostTest(TestCase):
"""Test cases for queueing outgoing webmentions during feed scans."""
def setUp(self):
"""Set up test fixtures."""
self.feed_url = "https://example.com/social.org"
self.feed = Feed.objects.create(url=self.feed_url)
self.profile = Profile.objects.create(
feed=self.feed_url, nick="alice", title="Alice"
)
self.post_id = "2026-01-01T10:00:00+00:00"
def test_queues_external_urls_as_pending(self):
# Given: A post linking to an external article
content = "I wrote about it: [[https://blog.example.org/post][my post]]"
# When: Webmentions are queued for the post
queued = queue_webmentions_for_post(self.profile, self.post_id, content)
# Then: One pending webmention exists with the post as source
self.assertEqual(queued, 1)
webmention = OutgoingWebmention.objects.get()
self.assertEqual(webmention.source, f"{self.feed_url}#{self.post_id}")
self.assertEqual(webmention.target, "https://blog.example.org/post")
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_PENDING)
def test_rescanning_same_post_does_not_duplicate(self):
# Given: A post whose webmention was already queued
content = "Look at https://blog.example.org/post please"
queue_webmentions_for_post(self.profile, self.post_id, content)
# When: The same post is queued again (rescan)
queued = queue_webmentions_for_post(self.profile, self.post_id, content)
# Then: Nothing new is queued
self.assertEqual(queued, 0)
self.assertEqual(OutgoingWebmention.objects.count(), 1)
def test_rescan_does_not_reset_sent_status(self):
# Given: A webmention already delivered for the post
content = "Look at https://blog.example.org/post please"
queue_webmentions_for_post(self.profile, self.post_id, content)
OutgoingWebmention.objects.update(status=OutgoingWebmention.STATUS_SENT)
# When: The same post is queued again (rescan)
queue_webmentions_for_post(self.profile, self.post_id, content)
# Then: The delivered webmention keeps its sent status
self.assertEqual(
OutgoingWebmention.objects.get().status, OutgoingWebmention.STATUS_SENT
)
def test_edited_post_queues_only_new_urls(self):
# Given: A post whose original link was already delivered
queue_webmentions_for_post(
self.profile, self.post_id, "First link https://blog.example.org/a"
)
OutgoingWebmention.objects.update(status=OutgoingWebmention.STATUS_SENT)
# When: The post is edited adding a second link
queued = queue_webmentions_for_post(
self.profile,
self.post_id,
"First link https://blog.example.org/a and https://blog.example.org/b",
)
# Then: Only the added link is queued, the delivered one is untouched
self.assertEqual(queued, 1)
self.assertEqual(OutgoingWebmention.objects.count(), 2)
self.assertEqual(
OutgoingWebmention.objects.get(target="https://blog.example.org/a").status,
OutgoingWebmention.STATUS_SENT,
)
def test_registered_feeds_are_excluded(self):
# Given: A post linking to a registered Org Social feed and to an article
Feed.objects.create(url="https://friend.example.org/social.org")
content = (
"Read https://friend.example.org/social.org#2026-01-01T00:00:00+00:00 "
"and https://blog.example.org/post"
)
# When: Webmentions are queued for the post
queued = queue_webmentions_for_post(self.profile, self.post_id, content)
# Then: Only the article is queued, the feed link is handled natively
self.assertEqual(queued, 1)
self.assertEqual(
OutgoingWebmention.objects.get().target, "https://blog.example.org/post"
)
def test_own_feed_is_excluded(self):
# Given: A post linking to the author's own feed
content = f"Self reference {self.feed_url}#2025-12-31T00:00:00+00:00"
# When: Webmentions are queued for the post
queued = queue_webmentions_for_post(self.profile, self.post_id, content)
# Then: Nothing is queued
self.assertEqual(queued, 0)
def test_post_without_urls_queues_nothing(self):
# Given: A post with no URLs at all
content = "Just text"
# When: Webmentions are queued for the post
queued = queue_webmentions_for_post(self.profile, self.post_id, content)
# Then: Nothing is queued
self.assertEqual(queued, 0)
self.assertEqual(OutgoingWebmention.objects.count(), 0)
class ScanFeedsWebmentionIntegrationTest(TestCase):
"""Integration tests: scan_feeds queues webmentions exactly once."""
def setUp(self):
"""Set up test fixtures."""
self.feed_url = "https://example.com/social.org"
self.feed = Feed.objects.create(url=self.feed_url)
def _parsed_feed(self, content):
return {
"metadata": {"title": "Alice", "nick": "alice"},
"posts": [
{
"id": "2026-01-01T10:00:00+00:00",
"content": content,
"properties": {},
"mentions": [],
"poll_options": [],
}
],
}
@patch("app.feeds.parser.parse_org_social")
def test_new_post_queues_webmention_once(self, mock_parse):
from app.feeds.tasks import scan_feeds
# Given: A feed with a new post linking to an article
mock_parse.return_value = self._parsed_feed(
"Article: https://blog.example.org/post"
)
# When: The feed is scanned twice without changes
scan_feeds.call_local()
first_scan_count = OutgoingWebmention.objects.count()
scan_feeds.call_local()
# Then: Exactly one webmention is queued, the rescan adds nothing
self.assertEqual(first_scan_count, 1)
self.assertEqual(OutgoingWebmention.objects.count(), 1)
@patch("app.feeds.parser.parse_org_social")
def test_edited_post_queues_only_added_link(self, mock_parse):
from app.feeds.tasks import scan_feeds
# Given: A scanned feed whose post links to one article
mock_parse.return_value = self._parsed_feed(
"Article: https://blog.example.org/a"
)
scan_feeds.call_local()
self.assertEqual(OutgoingWebmention.objects.count(), 1)
# When: The post is edited adding a second link and rescanned
mock_parse.return_value = self._parsed_feed(
"Article: https://blog.example.org/a and https://blog.example.org/b"
)
scan_feeds.call_local()
# Then: Only the added link produced a new webmention
self.assertEqual(OutgoingWebmention.objects.count(), 2)
targets = set(OutgoingWebmention.objects.values_list("target", flat=True))
self.assertEqual(
targets, {"https://blog.example.org/a", "https://blog.example.org/b"}
)
class SendPendingWebmentionsTest(TestCase):
"""Test cases for the delivery task."""
def _create_webmention(self, **kwargs):
defaults = {
"source": "https://example.com/social.org#2026-01-01T10:00:00+00:00",
"target": "https://blog.example.org/post",
}
defaults.update(kwargs)
return OutgoingWebmention.objects.create(**defaults)
@patch("app.feeds.webmentions.is_safe_endpoint", return_value=True)
@patch("app.feeds.webmentions.send_webmention")
@patch("app.feeds.webmentions.discover_webmention_endpoint")
def test_pending_webmention_is_sent(self, mock_discover, mock_send, _mock_safe):
# Given: A pending webmention whose target accepts it with 202
mock_discover.return_value = "https://blog.example.org/wm"
mock_send.return_value = 202
webmention = self._create_webmention()
# When: The delivery task runs
counters = _send_pending_webmentions_impl()
# Then: The webmention is delivered once and marked as sent
self.assertEqual(counters["sent"], 1)
webmention.refresh_from_db()
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_SENT)
self.assertEqual(webmention.endpoint, "https://blog.example.org/wm")
self.assertEqual(webmention.response_code, 202)
self.assertEqual(webmention.attempts, 1)
mock_send.assert_called_once_with(
"https://blog.example.org/wm", webmention.source, webmention.target
)
@patch("app.feeds.webmentions.discover_webmention_endpoint")
def test_target_without_endpoint_is_marked_permanently(self, mock_discover):
# Given: A pending webmention whose target has no endpoint
mock_discover.return_value = None
webmention = self._create_webmention()
# When: The delivery task runs
counters = _send_pending_webmentions_impl()
# Then: The webmention is marked permanently as no_endpoint
self.assertEqual(counters["no_endpoint"], 1)
webmention.refresh_from_db()
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_NO_ENDPOINT)
# Then: A later run never fetches that target again
mock_discover.reset_mock()
_send_pending_webmentions_impl()
mock_discover.assert_not_called()
@patch("app.feeds.webmentions.is_safe_endpoint", return_value=False)
@patch("app.feeds.webmentions.discover_webmention_endpoint")
def test_unsafe_endpoint_is_rejected(self, mock_discover, _mock_safe):
# Given: A pending webmention whose endpoint points to loopback
mock_discover.return_value = "http://127.0.0.1/wm"
webmention = self._create_webmention()
# When: The delivery task runs
counters = _send_pending_webmentions_impl()
# Then: Nothing is sent and the webmention is discarded
self.assertEqual(counters["no_endpoint"], 1)
webmention.refresh_from_db()
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_NO_ENDPOINT)
@patch("app.feeds.webmentions.discover_webmention_endpoint")
def test_network_error_marks_failed_and_retries_later(self, mock_discover):
# Given: A pending webmention whose target is unreachable
mock_discover.side_effect = requests.RequestException("boom")
webmention = self._create_webmention()
# When: The delivery task runs
counters = _send_pending_webmentions_impl()
# Then: The webmention is marked failed with one attempt
self.assertEqual(counters["failed"], 1)
webmention.refresh_from_db()
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_FAILED)
self.assertEqual(webmention.attempts, 1)
# Then: Within the backoff window it is skipped
mock_discover.reset_mock()
_send_pending_webmentions_impl()
mock_discover.assert_not_called()
# Then: After the backoff window it is retried
webmention.last_attempt_at = timezone.now() - timedelta(hours=2)
webmention.save()
mock_discover.side_effect = None
mock_discover.return_value = None
_send_pending_webmentions_impl()
webmention.refresh_from_db()
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_NO_ENDPOINT)
@patch("app.feeds.webmentions.discover_webmention_endpoint")
def test_max_attempts_reached_stops_retrying(self, mock_discover):
# Given: A webmention that already exhausted its attempts long ago
self._create_webmention(
status=OutgoingWebmention.STATUS_FAILED,
attempts=WEBMENTION_MAX_ATTEMPTS,
last_attempt_at=timezone.now() - timedelta(days=30),
)
# When: The delivery task runs
counters = _send_pending_webmentions_impl()
# Then: It is never processed again
self.assertEqual(counters, {"sent": 0, "no_endpoint": 0, "failed": 0})
mock_discover.assert_not_called()
@patch("app.feeds.webmentions.is_safe_endpoint", return_value=True)
@patch("app.feeds.webmentions.send_webmention")
@patch("app.feeds.webmentions.discover_webmention_endpoint")
def test_http_error_response_marks_failed(
self, mock_discover, mock_send, _mock_safe
):
# Given: A pending webmention whose endpoint answers HTTP 500
mock_discover.return_value = "https://blog.example.org/wm"
mock_send.return_value = 500
webmention = self._create_webmention()
# When: The delivery task runs
counters = _send_pending_webmentions_impl()
# Then: The webmention is marked failed keeping the endpoint for retry
self.assertEqual(counters["failed"], 1)
webmention.refresh_from_db()
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_FAILED)
self.assertEqual(webmention.response_code, 500)
self.assertEqual(webmention.endpoint, "https://blog.example.org/wm")

View file

@ -4,6 +4,8 @@ from rest_framework import status
from django.core.cache import cache from django.core.cache import cache
import logging import logging
from app.bridge.models import bridge_urls_q, is_bridge_feed_url
from .models import Feed from .models import Feed
from .parser import validate_org_social_feed from .parser import validate_org_social_feed
@ -32,8 +34,11 @@ class FeedsView(APIView):
status=status.HTTP_200_OK, status=status.HTTP_200_OK,
) )
# If not in cache, query database # If not in cache, query database. Bridge virtual feeds are a
feeds = list(Feed.objects.all().values_list("url", flat=True)) # connection helper, not real accounts, so they are never listed.
feeds = list(
Feed.objects.exclude(bridge_urls_q("url")).values_list("url", flat=True)
)
# Cache permanently (will be cleared by scan_feeds task) # Cache permanently (will be cleared by scan_feeds task)
cache.set(cache_key, feeds, None) cache.set(cache_key, feeds, None)
@ -62,6 +67,18 @@ class FeedsView(APIView):
feed_url = feed_url.strip() feed_url = feed_url.strip()
# Bridge virtual feeds serve valid Org Social content but are not
# real accounts; they cannot be registered
if is_bridge_feed_url(feed_url):
return Response(
{
"type": "Error",
"errors": ["Bridge virtual feeds cannot be registered"],
"data": None,
},
status=status.HTTP_400_BAD_REQUEST,
)
# Check if feed already exists # Check if feed already exists
existing_feed = Feed.objects.filter(url=feed_url).first() existing_feed = Feed.objects.filter(url=feed_url).first()
if existing_feed: if existing_feed:

202
app/feeds/webmentions.py Normal file
View file

@ -0,0 +1,202 @@
"""
Outgoing Webmention support (sender side only).
Implements the sender half of https://www.w3.org/TR/webmention/:
URL extraction from post content, endpoint discovery and notification.
Receiving webmentions is out of scope for the relay.
"""
import ipaddress
import logging
import re
import socket
from html.parser import HTMLParser
from urllib.parse import urljoin, urlparse
import requests
logger = logging.getLogger(__name__)
WEBMENTION_TIMEOUT = 10
WEBMENTION_USER_AGENT = "Org-Social-Relay (+https://relay.org-social.org/)"
# Maximum HTML bytes read while looking for a <link>/<a> endpoint
DISCOVERY_MAX_BYTES = 1024 * 1024
# Org Social mentions ([[org-social:url][nick]]) are handled natively by the
# relay and must never produce webmentions
ORG_SOCIAL_MENTION_RE = re.compile(r"\[\[org-social:[^\]]*\](?:\[[^\]]*\])?\]")
ORG_LINK_RE = re.compile(r"\[\[(https?://[^\]\[]+)\](?:\[[^\]]*\])?\]")
BARE_URL_RE = re.compile(r"https?://[^\s\]\[<>\"']+")
TRAILING_PUNCTUATION = ".,;:!?)'\""
def extract_external_urls(content):
"""
Extract http(s) URLs from post content, both Org links ([[url]] and
[[url][description]]) and bare URLs. Org Social mentions are ignored.
Returns a list of unique URLs in order of appearance.
"""
if not content:
return []
text = ORG_SOCIAL_MENTION_RE.sub(" ", content)
urls = []
for match in ORG_LINK_RE.finditer(text):
urls.append(match.group(1).strip())
# Remove Org link syntax so bare URL matching does not see them again
text = ORG_LINK_RE.sub(" ", text)
for match in BARE_URL_RE.finditer(text):
urls.append(match.group(0).rstrip(TRAILING_PUNCTUATION))
unique_urls = []
seen = set()
for url in urls:
if url and url not in seen:
seen.add(url)
unique_urls.append(url)
return unique_urls
class _EndpointHTMLParser(HTMLParser):
"""Finds the first <link> or <a> with rel="webmention" in document order."""
def __init__(self):
super().__init__(convert_charrefs=True)
self.endpoint = None
def handle_starttag(self, tag, attrs):
if self.endpoint is not None or tag not in ("link", "a"):
return
attrs_dict = dict(attrs)
rel = attrs_dict.get("rel") or ""
if "webmention" in rel.lower().split() and "href" in attrs_dict:
# An empty href is valid: it resolves to the page URL itself
self.endpoint = attrs_dict.get("href") or ""
def discover_webmention_endpoint(target_url):
"""
Discover the Webmention endpoint of a target URL following the spec
precedence: first HTTP Link header, then first <link>/<a> element in
document order. Relative endpoints are resolved against the final URL
after redirects.
Returns the absolute endpoint URL, or None if the target does not
advertise one. Raises requests.RequestException on network errors.
"""
response = requests.get(
target_url,
timeout=WEBMENTION_TIMEOUT,
headers={"User-Agent": WEBMENTION_USER_AGENT},
stream=True,
)
try:
link_header = response.headers.get("Link", "")
if link_header:
for link in requests.utils.parse_header_links(link_header):
rels = link.get("rel", "").lower().split()
if "webmention" in rels and "url" in link:
return urljoin(response.url, link["url"])
content_type = response.headers.get("Content-Type", "")
if "html" not in content_type.lower():
return None
raw_content = response.raw.read(DISCOVERY_MAX_BYTES, decode_content=True)
html = raw_content.decode(response.encoding or "utf-8", errors="replace")
finally:
response.close()
parser = _EndpointHTMLParser()
parser.feed(html)
if parser.endpoint is None:
return None
return urljoin(response.url, parser.endpoint)
def is_safe_endpoint(endpoint_url):
"""
Reject endpoints that are not plain http(s) or that resolve to loopback,
private, link-local or otherwise non-global addresses (spec section 4.3).
"""
parsed = urlparse(endpoint_url)
if parsed.scheme not in ("http", "https") or not parsed.hostname:
return False
try:
addr_info = socket.getaddrinfo(parsed.hostname, None)
except (socket.gaierror, UnicodeError):
return False
for info in addr_info:
try:
address = ipaddress.ip_address(info[4][0])
except ValueError:
return False
if (
address.is_loopback
or address.is_private
or address.is_link_local
or address.is_multicast
or address.is_reserved
or address.is_unspecified
):
return False
return True
def send_webmention(endpoint_url, source, target):
"""
POST the webmention as application/x-www-form-urlencoded and return the
HTTP status code. Raises requests.RequestException on network errors.
"""
response = requests.post(
endpoint_url,
data={"source": source, "target": target},
timeout=WEBMENTION_TIMEOUT,
headers={"User-Agent": WEBMENTION_USER_AGENT},
)
return response.status_code
def queue_webmentions_for_post(profile, post_id, content):
"""
Create pending OutgoingWebmention rows for every external URL in a post.
URLs pointing to registered Org Social feeds (or the author's own feed)
are skipped: the relay already handles those interactions natively.
Existing (source, target) pairs are never touched, so a webmention is
sent at most once even across rescans and post edits.
Returns the number of newly queued webmentions.
"""
from .models import Feed, OutgoingWebmention
urls = extract_external_urls(content)
if not urls:
return 0
source = f"{profile.feed}#{post_id}"
base_urls = {url.split("#")[0] for url in urls}
known_feeds = set(
Feed.objects.filter(url__in=base_urls).values_list("url", flat=True)
)
queued = 0
for url in urls:
base_url = url.split("#")[0]
if base_url == profile.feed or base_url in known_feeds:
continue
_, created = OutgoingWebmention.objects.get_or_create(source=source, target=url)
if created:
queued += 1
if queued:
logger.info(f"Queued {queued} webmention(s) for {source}")
return queued

0
app/profile/__init__.py Normal file
View file

6
app/profile/apps.py Normal file
View file

@ -0,0 +1,6 @@
from django.apps import AppConfig
class ProfileConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "app.profile"

176
app/profile/tests.py Normal file
View file

@ -0,0 +1,176 @@
from django.test import TestCase
from rest_framework.test import APIClient
from rest_framework import status
from app.feeds.models import Follow, Profile
class ProfileViewTest(TestCase):
"""Test cases for the ProfileView API using Given/When/Then structure."""
def setUp(self):
self.client = APIClient()
self.url = "/profile/"
self.profile_main = Profile.objects.create(
feed="https://example.com/social.org",
title="Example Profile",
nick="example_user",
)
self.profile_alice = Profile.objects.create(
feed="https://alice.org/social.org",
title="Alice",
nick="alice",
)
self.profile_bob = Profile.objects.create(
feed="https://bob.org/social.org",
title="Bob",
nick="bob",
)
self.profile_carol = Profile.objects.create(
feed="https://carol.org/social.org",
title="Carol",
nick="carol",
)
Follow.objects.create(follower=self.profile_alice, followed=self.profile_main)
Follow.objects.create(follower=self.profile_bob, followed=self.profile_main)
def test_get_profile_with_followers(self):
"""Test GET /profile/?feed=<url> returns correct followers list."""
# Given: A profile followed by two users
feed_url = self.profile_main.feed
# When: We request the profile
response = self.client.get(self.url, {"feed": feed_url})
# Then: Response is successful
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["type"], "Success")
self.assertEqual(response.data["errors"], [])
# Then: Data contains feed and followers
data = response.data["data"]
self.assertEqual(data["feed"], feed_url)
self.assertIsInstance(data["followers"], list)
self.assertIn(self.profile_alice.feed, data["followers"])
self.assertIn(self.profile_bob.feed, data["followers"])
self.assertNotIn(self.profile_carol.feed, data["followers"])
# Then: Meta contains correct counts
meta = response.data["meta"]
self.assertEqual(meta["feed"], feed_url)
self.assertEqual(meta["total_followers"], 2)
def test_get_profile_no_followers(self):
"""Test GET /profile/ returns empty followers for profile with no followers."""
# Given: A profile with no followers
feed_url = self.profile_carol.feed
# When: We request the profile
response = self.client.get(self.url, {"feed": feed_url})
# Then: Response is successful with empty followers
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["type"], "Success")
self.assertEqual(response.data["data"]["followers"], [])
self.assertEqual(response.data["meta"]["total_followers"], 0)
def test_get_profile_not_found(self):
"""Test GET /profile/ returns 404 for non-existent feed."""
# Given: A feed URL not registered in the relay
nonexistent_feed = "https://nonexistent.com/social.org"
# When: We request the profile
response = self.client.get(self.url, {"feed": nonexistent_feed})
# Then: We get 404
self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)
self.assertEqual(response.data["type"], "Error")
self.assertIn("Profile not found", response.data["errors"][0])
self.assertIsNone(response.data["data"])
def test_get_profile_missing_feed_param(self):
"""Test GET /profile/ returns 400 when feed param is missing."""
# When: We request without the feed parameter
response = self.client.get(self.url)
# Then: We get 400
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertEqual(response.data["type"], "Error")
self.assertIn("required", response.data["errors"][0])
def test_get_profile_response_format(self):
"""Test GET /profile/ response matches README specification."""
# Given: A registered profile
feed_url = self.profile_main.feed
# When: We request the profile
response = self.client.get(self.url, {"feed": feed_url})
# Then: Response keys are correct
self.assertIn("type", response.data)
self.assertIn("errors", response.data)
self.assertIn("data", response.data)
self.assertIn("meta", response.data)
self.assertIn("_links", response.data)
data = response.data["data"]
self.assertIn("feed", data)
self.assertIn("followers", data)
meta = response.data["meta"]
self.assertIn("feed", meta)
self.assertIn("total_followers", meta)
links = response.data["_links"]
self.assertIn("self", links)
self.assertIn("href", links["self"])
self.assertIn("method", links["self"])
self.assertEqual(links["self"]["method"], "GET")
def test_get_profile_self_link_encoded(self):
"""Test that the self link URL-encodes the feed parameter."""
# Given: A feed URL with special characters
feed_url = self.profile_main.feed
# When: We request the profile
response = self.client.get(self.url, {"feed": feed_url})
# Then: Self link is URL-encoded
self.assertEqual(response.status_code, status.HTTP_200_OK)
href = response.data["_links"]["self"]["href"]
self.assertIn("%3A", href) # : is encoded
self.assertIn("%2F", href) # / is encoded
def test_get_profile_http_methods(self):
"""Test that only GET is allowed on the profile endpoint."""
# Given: A valid feed URL
params = {"feed": self.profile_main.feed}
# When/Then: Non-GET methods return 405
self.assertEqual(
self.client.post(self.url, params).status_code,
status.HTTP_405_METHOD_NOT_ALLOWED,
)
self.assertEqual(
self.client.put(self.url, params).status_code,
status.HTTP_405_METHOD_NOT_ALLOWED,
)
self.assertEqual(
self.client.delete(self.url, params).status_code,
status.HTTP_405_METHOD_NOT_ALLOWED,
)
def test_get_profile_followers_are_feeds_only(self):
"""Test that followers list contains only feed URLs, not profile metadata."""
# Given: A profile with a follower
feed_url = self.profile_main.feed
# When: We request the profile
response = self.client.get(self.url, {"feed": feed_url})
# Then: Each follower entry is a plain URL string
for follower in response.data["data"]["followers"]:
self.assertIsInstance(follower, str)
self.assertTrue(follower.startswith("http"))

7
app/profile/urls.py Normal file
View file

@ -0,0 +1,7 @@
from django.urls import path
from app.profile.views import ProfileView
urlpatterns = [
path("", ProfileView.as_view(), name="profile"),
]

76
app/profile/views.py Normal file
View file

@ -0,0 +1,76 @@
from urllib.parse import quote
from rest_framework import status
from rest_framework.response import Response
from rest_framework.views import APIView
from django.core.cache import cache
from app.feeds.models import Follow, Profile
class ProfileView(APIView):
"""Get profile information including followers for a given feed URL."""
def get(self, request):
feed_url = request.query_params.get("feed")
if not feed_url or not feed_url.strip():
return Response(
{
"type": "Error",
"errors": ["Feed URL parameter is required"],
"data": None,
},
status=status.HTTP_400_BAD_REQUEST,
)
feed_url = feed_url.strip()
cache_key = f"profile_{feed_url}"
cached_response = cache.get(cache_key)
if cached_response is not None:
return Response(cached_response, status=status.HTTP_200_OK)
try:
profile = Profile.objects.get(feed=feed_url)
except Profile.DoesNotExist:
return Response(
{
"type": "Error",
"errors": ["Profile not found for the given feed URL"],
"data": None,
},
status=status.HTTP_404_NOT_FOUND,
)
followers = (
Follow.objects.filter(followed=profile)
.select_related("follower")
.order_by("follower__feed")
)
follower_feeds = [f.follower.feed for f in followers]
encoded_feed_url = quote(feed_url, safe="")
response_data = {
"type": "Success",
"errors": [],
"data": {
"feed": feed_url,
"followers": follower_feeds,
},
"meta": {
"feed": feed_url,
"total_followers": len(follower_feeds),
},
"_links": {
"self": {
"href": f"/profile/?feed={encoded_feed_url}",
"method": "GET",
}
},
}
cache.set(cache_key, response_data, None)
return Response(response_data, status=status.HTTP_200_OK)

View file

@ -40,6 +40,10 @@ def root_view(request):
"method": "GET", "method": "GET",
"templated": True, "templated": True,
}, },
"sse-notifications-all": {
"href": "/sse/notifications/",
"method": "GET",
},
"reactions": { "reactions": {
"href": "/reactions/?feed={feed_url}", "href": "/reactions/?feed={feed_url}",
"method": "GET", "method": "GET",
@ -82,11 +86,30 @@ def root_view(request):
"method": "GET", "method": "GET",
"templated": True, "templated": True,
}, },
"profile": {
"href": "/profile/?feed={feed_url}",
"method": "GET",
"templated": True,
},
"rss": { "rss": {
"href": "/rss.xml", "href": "/rss.xml",
"method": "GET", "method": "GET",
"description": "RSS feed of latest posts (supports ?tag={tag} and ?feed={feed_url} filters)", "description": "RSS feed of latest posts (supports ?tag={tag} and ?feed={feed_url} filters)",
}, },
"stats": {"href": "/stats/", "method": "GET"},
"bridge": {"href": "/bridge/", "method": "GET"},
"bridge-activitypub": {
"href": "/bridge/activitypub/@{user}@{instance}/",
"method": "GET",
"templated": True,
"description": "ActivityPub account as a virtual social.org feed",
},
"bridge-rss": {
"href": "/bridge/rss/?url={feed_url}",
"method": "GET",
"templated": True,
"description": "RSS/Atom feed as a virtual social.org feed",
},
}, },
} }
) )

View file

@ -425,6 +425,89 @@ class RSSFeedTest(TestCase):
# After HTML conversion, should have actual content # After HTML conversion, should have actual content
self.assertIn("<p>", description.text) self.assertIn("<p>", description.text)
def test_rss_feed_excludes_bridge_posts(self):
"""Test that posts from bridge virtual feeds are excluded from the feed."""
# Given: A bridge profile with a post, alongside the regular posts
bridge_profile = Profile.objects.create(
feed=(
"https://relay.org-social.org/bridge/rss/"
"?url=https%3A%2F%2Fexample.org%2Ffeed.xml"
),
title="Bridged Feed",
nick="bridged_feed",
description="Bridge virtual feed",
)
bridge_post = Post.objects.create(
profile=bridge_profile,
post_id="2025-01-01T20:00:00+00:00",
content="Post coming from a bridged RSS feed",
tags="bridged",
)
# When: We request the global RSS feed
response = self.client.get(self.rss_url)
# Then: Only the posts from real profiles are included
root = ET.fromstring(response.content)
items = root.find("channel").findall("item")
self.assertEqual(len(items), 5)
guids = [item.find("guid").text for item in items]
for guid in guids:
self.assertNotIn(bridge_post.post_id, guid)
def test_rss_feed_filtered_by_tag_excludes_bridge_posts(self):
"""Test that bridge posts are excluded from tag-filtered feeds too."""
# Given: A bridge profile with a post tagged like a regular post
bridge_profile = Profile.objects.create(
feed="https://relay.org-social.org/bridge/activitypub/@user@instance.tld/",
title="Bridged Account",
nick="bridged_account",
description="Bridge virtual feed",
)
Post.objects.create(
profile=bridge_profile,
post_id="2025-01-01T21:00:00+00:00",
content="Bridged post about Emacs",
tags="emacs",
)
# When: We request the RSS feed filtered by tag "emacs"
response = self.client.get(self.rss_url, {"tag": "emacs"})
# Then: Only the real posts with that tag are included
root = ET.fromstring(response.content)
items = root.find("channel").findall("item")
self.assertEqual(len(items), 2) # post1 and post4
def test_rss_feed_explicit_bridge_feed_is_still_served(self):
"""Test that explicitly requesting a bridge feed still returns its posts."""
# Given: A bridge profile with a post
bridge_feed_url = (
"https://relay.org-social.org/bridge/rss/"
"?url=https%3A%2F%2Fexample.org%2Ffeed.xml"
)
bridge_profile = Profile.objects.create(
feed=bridge_feed_url,
title="Bridged Feed",
nick="bridged_feed",
description="Bridge virtual feed",
)
bridge_post = Post.objects.create(
profile=bridge_profile,
post_id="2025-01-01T20:00:00+00:00",
content="Post coming from a bridged RSS feed",
tags="bridged",
)
# When: We request the RSS feed filtered by the bridge feed itself
response = self.client.get(self.rss_url, {"feed": bridge_feed_url})
# Then: The bridge post is served
root = ET.fromstring(response.content)
items = root.find("channel").findall("item")
self.assertEqual(len(items), 1)
self.assertIn(bridge_post.post_id, items[0].find("guid").text)
@override_settings( @override_settings(
CACHES={ CACHES={
"default": { "default": {

View file

@ -7,6 +7,7 @@ import hashlib
import re import re
import logging import logging
from app.bridge.models import bridge_urls_q
from app.feeds.models import Post from app.feeds.models import Post
try: try:
@ -136,10 +137,15 @@ class LatestPostsFeed(Feed):
# Search by specific tag (exact word match, case insensitive) # Search by specific tag (exact word match, case insensitive)
tag_escaped = re.escape(obj["tag"]) tag_escaped = re.escape(obj["tag"])
tag_pattern = rf"(^|[\s]){tag_escaped}([\s]|$)" tag_pattern = rf"(^|[\s]){tag_escaped}([\s]|$)"
posts_query = posts_query.filter(tags__iregex=tag_pattern) posts_query = posts_query.filter(tags__iregex=tag_pattern).exclude(
bridge_urls_q("profile__feed")
)
elif obj["feed"]: elif obj["feed"]:
# Filter by author feed # Filter by author feed. An explicitly requested bridge feed is
# still served: only aggregate feeds must leave bridges out.
posts_query = posts_query.filter(profile__feed=obj["feed"]) posts_query = posts_query.filter(profile__feed=obj["feed"])
else:
posts_query = posts_query.exclude(bridge_urls_q("profile__feed"))
# Fetch more than 200 to account for filtering whitespace-only posts # Fetch more than 200 to account for filtering whitespace-only posts
posts_raw = list(posts_query[:250]) posts_raw = list(posts_query[:250])

View file

@ -1,89 +1,187 @@
import asyncio
import json import json
import pytest import pytest
from unittest.mock import patch, MagicMock from unittest.mock import patch, AsyncMock, MagicMock
from django.test import TestCase, Client from django.test import TestCase, Client
from app.feeds.notification_publisher import publish_notification from app.feeds.notification_publisher import publish_notification
class TestSSENotificationsEndpoint(TestCase): def collect_stream(response):
"""Test the SSE notifications endpoint""" """Consume an async streaming response and return the decoded content."""
async def _collect():
chunks = []
async for chunk in response.streaming_content:
if isinstance(chunk, bytes):
chunks.append(chunk)
else:
chunks.append(chunk.encode("utf-8"))
return b"".join(chunks).decode("utf-8")
return asyncio.run(_collect())
def make_async_pubsub(messages=None, error=None):
"""Build an async pubsub mock that returns messages then stops."""
mock_pubsub = AsyncMock()
if error:
mock_pubsub.get_message.side_effect = error
elif messages is not None:
# Yield each message, then raise RedisError to break the loop
import redis.asyncio as aioredis
mock_pubsub.get_message.side_effect = messages + [aioredis.RedisError("done")]
else:
import redis.asyncio as aioredis
mock_pubsub.get_message.side_effect = aioredis.RedisError("done")
return mock_pubsub
class TestSSENotificationsEndpoint(TestCase):
def setUp(self): def setUp(self):
"""Set up test fixtures."""
self.client = Client() self.client = Client()
self.feed_url = "https://example.com/social.org" self.feed_url = "https://example.com/social.org"
def test_sse_endpoint_requires_feed_parameter(self): def _patch_redis(self, pubsub):
"""Test that SSE endpoint requires feed parameter""" mock_redis = AsyncMock()
mock_redis.pubsub = MagicMock(return_value=pubsub) # pubsub() is a sync call
return patch("app.sse_notifications.views.aioredis.Redis", return_value=mock_redis)
def test_sse_endpoint_without_feed_returns_global_stream(self):
"""Test that SSE endpoint without feed parameter returns global stream."""
# Given: A Redis pubsub with no pending messages
pubsub = make_async_pubsub()
with self._patch_redis(pubsub):
# When: The SSE endpoint is requested without a feed parameter
response = self.client.get("/sse/notifications/") response = self.client.get("/sse/notifications/")
self.assertEqual(response.status_code, 400)
# Then: An event stream response is returned
self.assertEqual(response.status_code, 200)
self.assertEqual(response["Content-Type"], "text/event-stream") self.assertEqual(response["Content-Type"], "text/event-stream")
def test_sse_endpoint_accepts_valid_feed(self): def test_sse_endpoint_accepts_valid_feed(self):
"""Test that SSE endpoint accepts valid feed parameter""" """Test that SSE endpoint accepts valid feed parameter."""
with patch("app.sse_notifications.views.redis.Redis") as mock_redis: # Given: A Redis pubsub with no pending messages
# Mock Redis pubsub pubsub = make_async_pubsub()
mock_pubsub = MagicMock()
mock_pubsub.listen.return_value = iter(
[{"type": "subscribe", "channel": f"notifications:{self.feed_url}"}]
)
mock_redis.return_value.pubsub.return_value = mock_pubsub
with self._patch_redis(pubsub):
# When: The SSE endpoint is requested for a specific feed
response = self.client.get("/sse/notifications/", {"feed": self.feed_url}) response = self.client.get("/sse/notifications/", {"feed": self.feed_url})
# Then: An event stream response is returned with anti-buffering headers
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
self.assertEqual(response["Content-Type"], "text/event-stream") self.assertEqual(response["Content-Type"], "text/event-stream")
self.assertEqual(response["Cache-Control"], "no-cache") self.assertEqual(response["Cache-Control"], "no-cache")
self.assertEqual(response["X-Accel-Buffering"], "no") self.assertEqual(response["X-Accel-Buffering"], "no")
self.assertEqual(response["Access-Control-Allow-Origin"], "*") self.assertEqual(response["Access-Control-Allow-Origin"], "*")
def test_sse_sends_connection_event(self): def test_sse_sends_connection_event_with_feed(self):
"""Test that SSE sends initial connection event""" """Test that per-feed SSE sends initial connection event with feed field."""
with patch("app.sse_notifications.views.redis.Redis") as mock_redis: # Given: A Redis pubsub with no pending messages
# Mock Redis pubsub pubsub = make_async_pubsub()
mock_pubsub = MagicMock()
mock_pubsub.listen.return_value = iter([{"type": "subscribe"}])
mock_redis.return_value.pubsub.return_value = mock_pubsub
with self._patch_redis(pubsub):
# When: The per-feed stream is requested and consumed
response = self.client.get("/sse/notifications/", {"feed": self.feed_url}) response = self.client.get("/sse/notifications/", {"feed": self.feed_url})
content = collect_stream(response)
# Get the streaming content # Then: The stream starts with a connection event naming the feed
content = b"".join(response.streaming_content).decode("utf-8")
# Check for connection event
self.assertIn("event: connected", content) self.assertIn("event: connected", content)
self.assertIn(f'"feed": "{self.feed_url}"', content) self.assertIn(f'"feed": "{self.feed_url}"', content)
self.assertIn('"status": "connected"', content) self.assertIn('"status": "connected"', content)
def test_sse_sends_connection_event_global(self):
"""Test that global SSE sends initial connection event without feed field."""
# Given: A Redis pubsub with no pending messages
pubsub = make_async_pubsub()
with self._patch_redis(pubsub):
# When: The global stream is requested and consumed
response = self.client.get("/sse/notifications/")
content = collect_stream(response)
# Then: The stream starts with a connection event without a feed field
self.assertIn("event: connected", content)
self.assertIn('"status": "connected"', content)
self.assertNotIn('"feed":', content)
def test_sse_receives_notification_from_redis(self): def test_sse_receives_notification_from_redis(self):
"""Test that SSE receives and forwards notifications from Redis""" """Test that per-feed SSE receives and forwards notifications from Redis."""
# Given: A Redis pubsub holding one mention notification
notification_data = { notification_data = {
"type": "mention", "type": "mention",
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000", "post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
} }
pubsub = make_async_pubsub(
with patch("app.sse_notifications.views.redis.Redis") as mock_redis: messages=[{"type": "message", "data": json.dumps(notification_data)}]
# Mock Redis pubsub
mock_pubsub = MagicMock()
mock_pubsub.listen.return_value = iter(
[
{"type": "subscribe"},
{"type": "message", "data": json.dumps(notification_data)},
]
) )
mock_redis.return_value.pubsub.return_value = mock_pubsub
with self._patch_redis(pubsub):
# When: The per-feed stream is requested and consumed
response = self.client.get("/sse/notifications/", {"feed": self.feed_url}) response = self.client.get("/sse/notifications/", {"feed": self.feed_url})
content = collect_stream(response)
# Get the streaming content # Then: The notification is forwarded as an SSE event
content = b"".join(response.streaming_content).decode("utf-8")
# Check for notification event
self.assertIn("event: notification", content) self.assertIn("event: notification", content)
self.assertIn('"type": "mention"', content) self.assertIn('"type": "mention"', content)
self.assertIn( self.assertIn("https://alice.com/social.org#2024-01-01T10:00:00+0000", content)
"https://alice.com/social.org#2024-01-01T10:00:00+0000", content
def test_global_sse_adds_target_feed_to_notifications(self):
"""Test that global SSE adds target_feed field extracted from channel name."""
# Given: A Redis pubsub holding one pattern message for a feed channel
notification_data = {
"type": "mention",
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
}
target_feed = "https://example.com/social.org"
pubsub = make_async_pubsub(
messages=[
{
"type": "pmessage",
"pattern": "notifications:*",
"channel": f"notifications:{target_feed}",
"data": json.dumps(notification_data),
}
]
) )
with self._patch_redis(pubsub):
# When: The global stream is requested and consumed
response = self.client.get("/sse/notifications/")
content = collect_stream(response)
# Then: The forwarded event includes the target_feed from the channel
self.assertIn("event: notification", content)
self.assertIn(f'"target_feed": "{target_feed}"', content)
self.assertIn('"type": "mention"', content)
def test_global_sse_uses_psubscribe(self):
"""Test that global SSE subscribes with psubscribe to all notification channels."""
# Given: A Redis pubsub with no pending messages
pubsub = make_async_pubsub()
with self._patch_redis(pubsub):
# When: The global stream is requested and consumed
response = self.client.get("/sse/notifications/")
collect_stream(response)
# Then: The view subscribed by pattern to every notification channel
pubsub.psubscribe.assert_awaited_once_with("notifications:*")
def test_per_feed_sse_uses_subscribe(self):
"""Test that per-feed SSE subscribes with subscribe to the specific channel."""
# Given: A Redis pubsub with no pending messages
pubsub = make_async_pubsub()
with self._patch_redis(pubsub):
# When: The per-feed stream is requested and consumed
response = self.client.get("/sse/notifications/", {"feed": self.feed_url})
collect_stream(response)
# Then: The view subscribed only to that feed's channel
pubsub.subscribe.assert_awaited_once_with(f"notifications:{self.feed_url}")
class TestNotificationPublisher(TestCase): class TestNotificationPublisher(TestCase):
"""Test the notification publisher module""" """Test the notification publisher module"""
@ -91,25 +189,25 @@ class TestNotificationPublisher(TestCase):
@patch("app.feeds.notification_publisher.redis.Redis") @patch("app.feeds.notification_publisher.redis.Redis")
def test_publish_mention_notification(self, mock_redis): def test_publish_mention_notification(self, mock_redis):
"""Test publishing a mention notification""" """Test publishing a mention notification"""
# Given: A working Redis connection and a mention to notify
mock_redis_instance = MagicMock() mock_redis_instance = MagicMock()
mock_redis.return_value = mock_redis_instance mock_redis.return_value = mock_redis_instance
target_feed = "https://bob.com/social.org" target_feed = "https://bob.com/social.org"
post_url = "https://alice.com/social.org#2024-01-01T10:00:00+0000" post_url = "https://alice.com/social.org#2024-01-01T10:00:00+0000"
# When: The mention notification is published
result = publish_notification( result = publish_notification(
target_feed_url=target_feed, notification_type="mention", post_url=post_url target_feed_url=target_feed, notification_type="mention", post_url=post_url
) )
# Then: One message is published on the target feed's channel
self.assertTrue(result) self.assertTrue(result)
mock_redis_instance.publish.assert_called_once() mock_redis_instance.publish.assert_called_once()
# Check the published data
call_args = mock_redis_instance.publish.call_args call_args = mock_redis_instance.publish.call_args
channel, data = call_args[0] channel, data = call_args[0]
self.assertEqual(channel, f"notifications:{target_feed}") self.assertEqual(channel, f"notifications:{target_feed}")
notification = json.loads(data) notification = json.loads(data)
self.assertEqual(notification["type"], "mention") self.assertEqual(notification["type"], "mention")
self.assertEqual(notification["post"], post_url) self.assertEqual(notification["post"], post_url)
@ -117,95 +215,80 @@ class TestNotificationPublisher(TestCase):
@patch("app.feeds.notification_publisher.redis.Redis") @patch("app.feeds.notification_publisher.redis.Redis")
def test_publish_reaction_notification_with_emoji(self, mock_redis): def test_publish_reaction_notification_with_emoji(self, mock_redis):
"""Test publishing a reaction notification with emoji""" """Test publishing a reaction notification with emoji"""
# Given: A working Redis connection and a reaction to notify
mock_redis_instance = MagicMock() mock_redis_instance = MagicMock()
mock_redis.return_value = mock_redis_instance mock_redis.return_value = mock_redis_instance
target_feed = "https://bob.com/social.org" # When: The reaction notification is published
post_url = "https://alice.com/social.org#2024-01-01T10:00:00+0000"
parent_post = "https://bob.com/social.org#2024-01-01T09:00:00+0000"
result = publish_notification( result = publish_notification(
target_feed_url=target_feed, target_feed_url="https://bob.com/social.org",
notification_type="reaction", notification_type="reaction",
post_url=post_url, post_url="https://alice.com/social.org#2024-01-01T10:00:00+0000",
emoji="", emoji="",
parent=parent_post, parent="https://bob.com/social.org#2024-01-01T09:00:00+0000",
) )
# Then: The published message carries the reaction type and emoji
self.assertTrue(result) self.assertTrue(result)
# Check the published data
call_args = mock_redis_instance.publish.call_args call_args = mock_redis_instance.publish.call_args
channel, data = call_args[0] notification = json.loads(call_args[0][1])
notification = json.loads(data)
self.assertEqual(notification["type"], "reaction") self.assertEqual(notification["type"], "reaction")
self.assertEqual(notification["emoji"], "") self.assertEqual(notification["emoji"], "")
self.assertEqual(notification["parent"], parent_post)
@patch("app.feeds.notification_publisher.redis.Redis") @patch("app.feeds.notification_publisher.redis.Redis")
def test_publish_reply_notification(self, mock_redis): def test_publish_reply_notification(self, mock_redis):
"""Test publishing a reply notification""" """Test publishing a reply notification"""
# Given: A working Redis connection and a reply to notify
mock_redis_instance = MagicMock() mock_redis_instance = MagicMock()
mock_redis.return_value = mock_redis_instance mock_redis.return_value = mock_redis_instance
target_feed = "https://bob.com/social.org" # When: The reply notification is published
post_url = "https://alice.com/social.org#2024-01-01T10:00:00+0000"
parent_post = "https://bob.com/social.org#2024-01-01T09:00:00+0000"
result = publish_notification( result = publish_notification(
target_feed_url=target_feed, target_feed_url="https://bob.com/social.org",
notification_type="reply", notification_type="reply",
post_url=post_url, post_url="https://alice.com/social.org#2024-01-01T10:00:00+0000",
parent=parent_post, parent="https://bob.com/social.org#2024-01-01T09:00:00+0000",
) )
# Then: The published message carries the reply type
self.assertTrue(result) self.assertTrue(result)
notification = json.loads(mock_redis_instance.publish.call_args[0][1])
call_args = mock_redis_instance.publish.call_args
channel, data = call_args[0]
notification = json.loads(data)
self.assertEqual(notification["type"], "reply") self.assertEqual(notification["type"], "reply")
self.assertEqual(notification["parent"], parent_post)
@patch("app.feeds.notification_publisher.redis.Redis") @patch("app.feeds.notification_publisher.redis.Redis")
def test_publish_boost_notification(self, mock_redis): def test_publish_boost_notification(self, mock_redis):
"""Test publishing a boost notification""" """Test publishing a boost notification"""
# Given: A working Redis connection and a boost to notify
mock_redis_instance = MagicMock() mock_redis_instance = MagicMock()
mock_redis.return_value = mock_redis_instance mock_redis.return_value = mock_redis_instance
target_feed = "https://bob.com/social.org" # When: The boost notification is published
post_url = "https://alice.com/social.org#2024-01-01T10:00:00+0000"
boosted_post = "https://bob.com/social.org#2024-01-01T09:00:00+0000"
result = publish_notification( result = publish_notification(
target_feed_url=target_feed, target_feed_url="https://bob.com/social.org",
notification_type="boost", notification_type="boost",
post_url=post_url, post_url="https://alice.com/social.org#2024-01-01T10:00:00+0000",
boosted=boosted_post, boosted="https://bob.com/social.org#2024-01-01T09:00:00+0000",
) )
# Then: The published message carries the boost type
self.assertTrue(result) self.assertTrue(result)
notification = json.loads(mock_redis_instance.publish.call_args[0][1])
call_args = mock_redis_instance.publish.call_args
channel, data = call_args[0]
notification = json.loads(data)
self.assertEqual(notification["type"], "boost") self.assertEqual(notification["type"], "boost")
self.assertEqual(notification["boosted"], boosted_post)
@patch("app.feeds.notification_publisher.redis.Redis") @patch("app.feeds.notification_publisher.redis.Redis")
def test_publish_notification_handles_redis_error(self, mock_redis): def test_publish_notification_handles_redis_error(self, mock_redis):
"""Test that publish_notification handles Redis errors gracefully""" """Test that publish_notification handles Redis errors gracefully"""
# Given: A Redis connection that fails
mock_redis.side_effect = Exception("Redis connection failed") mock_redis.side_effect = Exception("Redis connection failed")
# When: A notification is published
result = publish_notification( result = publish_notification(
target_feed_url="https://bob.com/social.org", target_feed_url="https://bob.com/social.org",
notification_type="mention", notification_type="mention",
post_url="https://alice.com/social.org#2024-01-01T10:00:00+0000", post_url="https://alice.com/social.org#2024-01-01T10:00:00+0000",
) )
# Then: The failure is reported without raising
self.assertFalse(result) self.assertFalse(result)
@ -214,20 +297,19 @@ class TestSSENotificationStructure:
"""Test that SSE notifications match the expected JSON structure""" """Test that SSE notifications match the expected JSON structure"""
def test_mention_notification_structure(self): def test_mention_notification_structure(self):
"""Test mention notification has correct structure""" # Given: A mention notification payload
notification = { notification = {
"type": "mention", "type": "mention",
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000", "post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
} }
# Verify structure # Then: It carries the mention type and a post URL with fragment
assert "type" in notification assert "type" in notification
assert "post" in notification
assert notification["type"] == "mention" assert notification["type"] == "mention"
assert "#" in notification["post"] assert "#" in notification["post"]
def test_reaction_notification_structure(self): def test_reaction_notification_structure(self):
"""Test reaction notification has correct structure""" # Given: A reaction notification payload
notification = { notification = {
"type": "reaction", "type": "reaction",
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000", "post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
@ -235,28 +317,43 @@ class TestSSENotificationStructure:
"parent": "https://bob.com/social.org#2024-01-01T09:00:00+0000", "parent": "https://bob.com/social.org#2024-01-01T09:00:00+0000",
} }
# Then: It carries the reaction type, emoji and parent
assert notification["type"] == "reaction" assert notification["type"] == "reaction"
assert "emoji" in notification assert "emoji" in notification
assert "parent" in notification assert "parent" in notification
def test_reply_notification_structure(self): def test_reply_notification_structure(self):
"""Test reply notification has correct structure""" # Given: A reply notification payload
notification = { notification = {
"type": "reply", "type": "reply",
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000", "post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
"parent": "https://bob.com/social.org#2024-01-01T09:00:00+0000", "parent": "https://bob.com/social.org#2024-01-01T09:00:00+0000",
} }
# Then: It carries the reply type and parent
assert notification["type"] == "reply" assert notification["type"] == "reply"
assert "parent" in notification assert "parent" in notification
def test_boost_notification_structure(self): def test_boost_notification_structure(self):
"""Test boost notification has correct structure""" # Given: A boost notification payload
notification = { notification = {
"type": "boost", "type": "boost",
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000", "post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
"boosted": "https://bob.com/social.org#2024-01-01T09:00:00+0000", "boosted": "https://bob.com/social.org#2024-01-01T09:00:00+0000",
} }
# Then: It carries the boost type and the boosted post
assert notification["type"] == "boost" assert notification["type"] == "boost"
assert "boosted" in notification assert "boosted" in notification
def test_global_notification_includes_target_feed(self):
# Given: A global stream notification payload
notification = {
"target_feed": "https://example.com/social.org",
"type": "mention",
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
}
# Then: It includes the target feed as an absolute URL
assert "target_feed" in notification
assert notification["target_feed"].startswith("http")

View file

@ -1,103 +1,63 @@
import asyncio
import json import json
import time import time
import logging import logging
from django.http import StreamingHttpResponse from django.http import StreamingHttpResponse
from django.views import View from django.views import View
from django.conf import settings from django.conf import settings
import redis
import redis.asyncio as aioredis
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
HEARTBEAT_INTERVAL = 30
def _sse_response(stream):
response = StreamingHttpResponse(stream, content_type="text/event-stream")
response["Cache-Control"] = "no-cache"
response["X-Accel-Buffering"] = "no"
response["Access-Control-Allow-Origin"] = "*"
response["Access-Control-Allow-Methods"] = "GET, OPTIONS"
response["Access-Control-Allow-Headers"] = "Content-Type"
return response
def _get_redis():
host = settings.HUEY["connection"]["host"]
port = settings.HUEY["connection"]["port"]
db = settings.HUEY["connection"]["db"]
return aioredis.Redis(host=host, port=port, db=db, decode_responses=True)
class SSENotificationsView(View): class SSENotificationsView(View):
""" """
Server-Sent Events (SSE) endpoint for real-time notifications. Server-Sent Events endpoint for real-time notifications.
Clients connect to this endpoint with a feed URL parameter and receive With ?feed=: streams notifications for a single feed.
real-time notifications as they are published by the scan_feeds task. Without ?feed=: streams all notifications from all feeds, adding target_feed to each event.
Usage:
GET /sse/notifications/?feed=https://example.com/social.org
Response format (Server-Sent Events):
event: notification
data: {"type": "mention", "post": "...", ...}
event: heartbeat
data: {"status": "alive"}
""" """
def get(self, request): async def get(self, request):
feed_url = request.GET.get("feed", "").strip() feed_url = request.GET.get("feed", "").strip()
stream = self._feed_stream(feed_url) if feed_url else self._global_stream()
return _sse_response(stream)
if not feed_url: async def _feed_stream(self, feed_url):
return StreamingHttpResponse( r = _get_redis()
"data: "
+ json.dumps({"error": "Feed URL parameter is required"})
+ "\n\n",
content_type="text/event-stream",
status=400,
)
logger.info(f"SSE connection established for feed: {feed_url}")
def event_stream():
"""Generator that yields SSE-formatted events"""
try:
# Connect to Redis
redis_host = settings.HUEY["connection"]["host"]
redis_port = settings.HUEY["connection"]["port"]
redis_db = settings.HUEY["connection"]["db"]
r = redis.Redis(
host=redis_host, port=redis_port, db=redis_db, decode_responses=True
)
# Subscribe to the feed's notification channel
pubsub = r.pubsub() pubsub = r.pubsub()
channel_name = f"notifications:{feed_url}" logger.info(f"SSE per-feed connection: {feed_url}")
pubsub.subscribe(channel_name) try:
await pubsub.subscribe(f"notifications:{feed_url}")
logger.info(f"Subscribed to Redis channel: {channel_name}")
# Send initial connection message
yield "event: connected\n" yield "event: connected\n"
yield f"data: {json.dumps({'feed': feed_url, 'status': 'connected'})}\n\n" yield f"data: {json.dumps({'feed': feed_url, 'status': 'connected'})}\n\n"
# Keep track of last heartbeat async for chunk in self._message_loop(pubsub):
last_heartbeat = time.time() yield chunk
heartbeat_interval = 30 # seconds
# Listen for messages with timeout for heartbeat except aioredis.RedisError as e:
for message in pubsub.listen(): logger.error(f"Redis error for {feed_url}: {e}")
# Send heartbeat every 30 seconds to keep connection alive
current_time = time.time()
if current_time - last_heartbeat >= heartbeat_interval:
yield "event: heartbeat\n"
yield f"data: {json.dumps({'status': 'alive', 'timestamp': int(current_time)})}\n\n"
last_heartbeat = current_time
# Process Redis messages
if message["type"] == "message":
try:
# Message data is already a JSON string from Redis
notification_data = json.loads(message["data"])
# Send notification event
yield "event: notification\n"
yield f"data: {json.dumps(notification_data)}\n\n"
logger.debug(
f"Sent notification to {feed_url}: {notification_data['type']}"
)
except json.JSONDecodeError as e:
logger.error(f"Failed to decode notification message: {e}")
except Exception as e:
logger.error(f"Error processing notification: {e}")
except redis.RedisError as e:
logger.error(f"Redis connection error for {feed_url}: {e}")
yield "event: error\n" yield "event: error\n"
yield f"data: {json.dumps({'error': 'Redis connection failed'})}\n\n" yield f"data: {json.dumps({'error': 'Redis connection failed'})}\n\n"
except Exception as e: except Exception as e:
@ -106,22 +66,79 @@ class SSENotificationsView(View):
yield f"data: {json.dumps({'error': 'Internal server error'})}\n\n" yield f"data: {json.dumps({'error': 'Internal server error'})}\n\n"
finally: finally:
try: try:
pubsub.close() await pubsub.aclose()
await r.aclose()
logger.info(f"SSE connection closed for feed: {feed_url}") logger.info(f"SSE connection closed for feed: {feed_url}")
except Exception: except Exception:
pass pass
response = StreamingHttpResponse( async def _global_stream(self):
event_stream(), content_type="text/event-stream" r = _get_redis()
pubsub = r.pubsub()
logger.info("SSE global connection established")
try:
await pubsub.psubscribe("notifications:*")
yield "event: connected\n"
yield f"data: {json.dumps({'status': 'connected'})}\n\n"
async for chunk in self._message_loop(pubsub, global_mode=True):
yield chunk
except aioredis.RedisError as e:
logger.error(f"Redis error in global SSE stream: {e}")
yield "event: error\n"
yield f"data: {json.dumps({'error': 'Redis connection failed'})}\n\n"
except Exception as e:
logger.error(f"Unexpected error in global SSE stream: {e}")
yield "event: error\n"
yield f"data: {json.dumps({'error': 'Internal server error'})}\n\n"
finally:
try:
await pubsub.aclose()
await r.aclose()
logger.info("Global SSE connection closed")
except Exception:
pass
async def _message_loop(self, pubsub, global_mode=False):
last_heartbeat = time.time()
while True:
current_time = time.time()
if current_time - last_heartbeat >= HEARTBEAT_INTERVAL:
yield "event: heartbeat\n"
yield f"data: {json.dumps({'status': 'alive', 'timestamp': int(current_time)})}\n\n"
last_heartbeat = current_time
message = await pubsub.get_message(
ignore_subscribe_messages=True, timeout=1.0
) )
# SSE headers if message is None:
response["Cache-Control"] = "no-cache" await asyncio.sleep(0)
response["X-Accel-Buffering"] = "no" # Disable nginx buffering continue
# CORS headers for cross-origin requests msg_type = message["type"]
response["Access-Control-Allow-Origin"] = "*" is_data = (msg_type == "message") or (
response["Access-Control-Allow-Methods"] = "GET, OPTIONS" global_mode and msg_type == "pmessage"
response["Access-Control-Allow-Headers"] = "Content-Type" )
if not is_data:
continue
return response try:
notification_data = json.loads(message["data"])
if global_mode:
channel = message["channel"]
notification_data["target_feed"] = channel.removeprefix(
"notifications:"
)
yield "event: notification\n"
yield f"data: {json.dumps(notification_data)}\n\n"
except json.JSONDecodeError as e:
logger.error(f"Failed to decode notification message: {e}")
except Exception as e:
logger.error(f"Error processing notification: {e}")

0
app/stats/__init__.py Normal file
View file

6
app/stats/apps.py Normal file
View file

@ -0,0 +1,6 @@
from django.apps import AppConfig
class StatsConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "app.stats"

295
app/stats/tests.py Normal file
View file

@ -0,0 +1,295 @@
from datetime import datetime, timezone as dt_timezone
from django.core.cache import cache
from django.test import TestCase, override_settings
from rest_framework import status
from rest_framework.test import APIClient
from app.feeds.models import Feed, Follow, Post, Profile
class StatsViewTest(TestCase):
"""Test cases for the StatsView API using Given/When/Then structure."""
def setUp(self):
self.client = APIClient()
self.stats_url = "/stats/"
cache.clear()
self.profile1 = Profile.objects.create(
feed="https://alice.org/social.org",
title="Alice",
nick="alice",
)
self.profile2 = Profile.objects.create(
feed="https://bob.org/social.org",
title="Bob",
nick="bob",
)
def tearDown(self):
cache.clear()
def _create_post(self, profile, post_id, **kwargs):
"""Create a post whose created_at matches its RFC 3339 post_id"""
created_at = datetime.fromisoformat(post_id).astimezone(dt_timezone.utc)
return Post.objects.create(
profile=profile,
post_id=post_id,
created_at=created_at,
**kwargs,
)
def test_empty_database_returns_empty_stats(self):
"""Test GET /stats/ with no posts returns empty years and zeroed globals."""
# Given: No posts, feeds or follows exist (only two profiles)
# When: We request the stats
response = self.client.get(self.stats_url)
# Then: The response is successful with empty monthly data
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["type"], "Success")
self.assertEqual(response.data["errors"], [])
self.assertEqual(response.data["data"]["years"], {})
self.assertEqual(
response.data["data"]["global"],
{
"registered_feeds": 0,
"total_accounts": 2,
"total_posts": 0,
"total_follows": 0,
"active_groups": 0,
},
)
self.assertIn("generated_at", response.data["meta"])
self.assertEqual(
response.data["_links"]["self"],
{"href": "/stats/", "method": "GET"},
)
def test_post_types_are_disjoint_and_sum_to_total(self):
"""Test each post type is counted once and the sum matches total_posts."""
# Given: One post of each type in the same month
self._create_post(self.profile1, "2025-03-01T10:00:00+00:00", content="Plain")
self._create_post(
self.profile1,
"2025-03-02T10:00:00+00:00",
content="A reply with text",
reply_to="https://bob.org/social.org#2025-03-01T09:00:00+00:00",
)
self._create_post(
self.profile1,
"2025-03-03T10:00:00+00:00",
content="",
reply_to="https://bob.org/social.org#2025-03-01T09:00:00+00:00",
mood="😄",
)
self._create_post(
self.profile1,
"2025-03-04T10:00:00+00:00",
content="",
include="https://bob.org/social.org#2025-03-01T09:00:00+00:00",
)
# When: We request the stats
response = self.client.get(self.stats_url)
# Then: Each type is counted once and the four counters sum to total
month = response.data["data"]["years"]["2025"]["03"]
self.assertEqual(month["total_posts"], 4)
self.assertEqual(month["posts"], 1)
self.assertEqual(month["replies"], 1)
self.assertEqual(month["reactions"], 1)
self.assertEqual(month["boosts"], 1)
self.assertEqual(
month["posts"] + month["replies"] + month["reactions"] + month["boosts"],
month["total_posts"],
)
def test_reaction_requires_mood_and_empty_content(self):
"""Test replies with mood and text count as replies, not reactions."""
# Given: A reply with mood but real content, and a reaction with
# whitespace-only content
self._create_post(
self.profile1,
"2025-05-01T10:00:00+00:00",
content="Great post!",
reply_to="https://bob.org/social.org#2025-04-30T09:00:00+00:00",
mood="🔥",
)
self._create_post(
self.profile2,
"2025-05-02T10:00:00+00:00",
content=" \n ",
reply_to="https://alice.org/social.org#2025-04-30T09:00:00+00:00",
mood="👍",
)
# When: We request the stats
response = self.client.get(self.stats_url)
# Then: Only the empty-content reply counts as a reaction
month = response.data["data"]["years"]["2025"]["05"]
self.assertEqual(month["replies"], 1)
self.assertEqual(month["reactions"], 1)
def test_group_messages_and_polls_are_transversal(self):
"""Test group messages and polls also count in their base type."""
# Given: A poll and a reply inside a group
self._create_post(
self.profile1,
"2025-06-01T10:00:00+00:00",
content="Cat or dog?",
poll_end=datetime(2025, 6, 10, tzinfo=dt_timezone.utc),
)
self._create_post(
self.profile2,
"2025-06-02T10:00:00+00:00",
content="I agree",
reply_to="https://alice.org/social.org#2025-06-01T09:00:00+00:00",
group="emacs",
)
# When: We request the stats
response = self.client.get(self.stats_url)
# Then: The poll counts as post and the group reply counts as reply
month = response.data["data"]["years"]["2025"]["06"]
self.assertEqual(month["total_posts"], 2)
self.assertEqual(month["posts"], 1)
self.assertEqual(month["replies"], 1)
self.assertEqual(month["polls"], 1)
self.assertEqual(month["group_messages"], 1)
def test_active_accounts_counts_distinct_profiles(self):
"""Test active_accounts counts each profile once per month."""
# Given: Two posts from the same profile and one from another
self._create_post(self.profile1, "2025-07-01T10:00:00+00:00", content="One")
self._create_post(self.profile1, "2025-07-02T10:00:00+00:00", content="Two")
self._create_post(self.profile2, "2025-07-03T10:00:00+00:00", content="Three")
# When: We request the stats
response = self.client.get(self.stats_url)
# Then: Only two distinct accounts are active
month = response.data["data"]["years"]["2025"]["07"]
self.assertEqual(month["active_accounts"], 2)
self.assertEqual(month["total_posts"], 3)
def test_posts_are_grouped_by_year_and_month(self):
"""Test posts land in the year/month of their ID with padded keys."""
# Given: Posts in different months and years
self._create_post(self.profile1, "2024-12-15T10:00:00+00:00", content="Dec")
self._create_post(self.profile1, "2025-01-15T10:00:00+00:00", content="Jan")
self._create_post(self.profile2, "2025-01-20T10:00:00+00:00", content="Jan 2")
# When: We request the stats
response = self.client.get(self.stats_url)
# Then: Months are grouped under their year with zero-padded keys
years = response.data["data"]["years"]
self.assertEqual(sorted(years.keys()), ["2024", "2025"])
self.assertEqual(years["2024"]["12"]["total_posts"], 1)
self.assertEqual(years["2025"]["01"]["total_posts"], 2)
def test_global_counters(self):
"""Test global counters reflect feeds, accounts, follows and groups."""
# Given: Feeds, follows and posts in two different groups
Feed.objects.create(url="https://alice.org/social.org")
Feed.objects.create(url="https://bob.org/social.org")
Feed.objects.create(url="https://carol.org/social.org")
Follow.objects.create(follower=self.profile1, followed=self.profile2)
self._create_post(
self.profile1, "2025-08-01T10:00:00+00:00", content="Hi", group="emacs"
)
self._create_post(
self.profile1, "2025-08-02T10:00:00+00:00", content="Hi", group="emacs"
)
self._create_post(
self.profile2, "2025-08-03T10:00:00+00:00", content="Hi", group="org-mode"
)
# When: We request the stats
response = self.client.get(self.stats_url)
# Then: Global counters match the created records
self.assertEqual(
response.data["data"]["global"],
{
"registered_feeds": 3,
"total_accounts": 2,
"total_posts": 3,
"total_follows": 1,
"active_groups": 2,
},
)
@override_settings(
CACHES={
"default": {
"BACKEND": "django.core.cache.backends.locmem.LocMemCache",
}
}
)
def test_response_is_cached(self):
"""Test the response is cached until the cache is cleared."""
# Given: A first request that populates the cache (DEBUG uses
# DummyCache, so a real in-memory backend is forced here)
self._create_post(self.profile1, "2025-09-01T10:00:00+00:00", content="One")
first = self.client.get(self.stats_url)
self.assertEqual(first.data["data"]["global"]["total_posts"], 1)
# When: A new post arrives without clearing the cache
self._create_post(self.profile1, "2025-09-02T10:00:00+00:00", content="Two")
cached = self.client.get(self.stats_url)
# Then: The cached response is returned until the cache is cleared
self.assertEqual(cached.data["data"]["global"]["total_posts"], 1)
cache.clear()
fresh = self.client.get(self.stats_url)
self.assertEqual(fresh.data["data"]["global"]["total_posts"], 2)
def test_bridge_feeds_do_not_count_in_stats(self):
"""Test bridge profiles, posts, feeds and follows are excluded."""
# Given: A real feed with posts and a bridge with feed, posts and follow
Feed.objects.create(url="https://alice.org/social.org")
self._create_post(self.profile1, "2025-08-01T10:00:00+00:00", content="Hi")
bridge_profile = Profile.objects.create(
feed=(
"https://relay.org-social.org/bridge/rss/"
"?url=https%3A%2F%2Frss.arxiv.org%2Frss%2Fquant-ph"
),
title="quant-ph updates on arXiv.org",
nick="quant-ph_updates_on_arXiv_org",
)
Feed.objects.create(url=bridge_profile.feed)
Follow.objects.create(follower=self.profile1, followed=bridge_profile)
for second in range(3):
self._create_post(
bridge_profile,
f"2025-08-02T04:00:0{second}+00:00",
content="Bridged paper",
group="physics",
)
# When: We request the stats
response = self.client.get(self.stats_url)
# Then: Global counters only reflect the real feed and its post
self.assertEqual(
response.data["data"]["global"],
{
"registered_feeds": 1,
"total_accounts": 2,
"total_posts": 1,
"total_follows": 0,
"active_groups": 0,
},
)
# Then: The monthly breakdown ignores the bridged posts
month = response.data["data"]["years"]["2025"]["08"]
self.assertEqual(month["total_posts"], 1)
self.assertEqual(month["active_accounts"], 1)

8
app/stats/urls.py Normal file
View file

@ -0,0 +1,8 @@
from django.urls import path
from .views import StatsView
app_name = "stats"
urlpatterns = [
path("", StatsView.as_view(), name="stats"),
]

117
app/stats/views.py Normal file
View file

@ -0,0 +1,117 @@
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework import status
from django.core.cache import cache
from django.db.models import Count, Q
from django.db.models.functions import TruncMonth
from django.utils import timezone
import logging
from app.bridge.models import bridge_urls_q
from app.feeds.models import Feed, Follow, Post, Profile
logger = logging.getLogger(__name__)
class StatsView(APIView):
"""Aggregated statistics grouped by year and month, plus global counters"""
def get(self, request):
cache_key = "stats"
cached_response = cache.get(cache_key)
if cached_response is not None:
return Response(cached_response, status=status.HTTP_200_OK)
response_data = {
"type": "Success",
"errors": [],
"data": {
"years": self._get_monthly_stats(),
"global": self._get_global_stats(),
},
"meta": {
"generated_at": timezone.now().isoformat(),
},
"_links": {
"self": {"href": "/stats/", "method": "GET"},
"feeds": {"href": "/feeds/", "method": "GET"},
},
}
# Cache permanently (will be cleared by scan_feeds task)
cache.set(cache_key, response_data, None)
return Response(response_data, status=status.HTTP_200_OK)
def _get_monthly_stats(self):
"""Aggregate post counters grouped by year and month.
A post belongs to the month of its ID (RFC 3339 timestamp), which
is stored parsed in created_at. The four type counters are disjoint
(posts + replies + reactions + boosts = total_posts), while
group_messages and polls are transversal.
"""
is_reply = ~Q(reply_to="")
# Same criterion as the replies endpoint: a reaction is a reply with
# a mood and empty or whitespace-only content
is_reaction = is_reply & ~Q(mood="") & Q(content__regex=r"^\s*$")
# Bridged posts belong to external authors, not to real accounts
monthly = (
Post.objects.exclude(bridge_urls_q("profile__feed"))
.annotate(month=TruncMonth("created_at"))
.values("month")
.annotate(
active_accounts=Count("profile", distinct=True),
total_posts=Count("id"),
posts=Count("id", filter=Q(reply_to="") & Q(include="")),
replies=Count("id", filter=is_reply & ~is_reaction),
boosts=Count("id", filter=Q(reply_to="") & ~Q(include="")),
reactions=Count("id", filter=is_reaction),
group_messages=Count("id", filter=~Q(group="")),
polls=Count("id", filter=Q(poll_end__isnull=False)),
)
.order_by("month")
)
years = {}
for row in monthly:
month = row["month"]
if month is None:
continue
year_key = f"{month.year:04d}"
month_key = f"{month.month:02d}"
years.setdefault(year_key, {})[month_key] = {
"active_accounts": row["active_accounts"],
"total_posts": row["total_posts"],
"posts": row["posts"],
"replies": row["replies"],
"boosts": row["boosts"],
"reactions": row["reactions"],
"group_messages": row["group_messages"],
"polls": row["polls"],
}
return years
def _get_global_stats(self):
"""Global counters, independent of the monthly breakdown.
Bridge virtual feeds (and their profiles, posts and follows) are
a connection helper, not real accounts, so they never count.
"""
real_posts = Post.objects.exclude(bridge_urls_q("profile__feed"))
return {
"registered_feeds": Feed.objects.exclude(bridge_urls_q("url")).count(),
"total_accounts": Profile.objects.exclude(bridge_urls_q("feed")).count(),
"total_posts": real_posts.count(),
"total_follows": Follow.objects.exclude(
bridge_urls_q("followed__feed")
).count(),
"active_groups": real_posts.exclude(group="")
.values("group")
.distinct()
.count(),
}

View file

@ -26,10 +26,11 @@ services:
- .:/app - .:/app
command: ./entrypoint.sh command: ./entrypoint.sh
healthcheck: healthcheck:
test: ["CMD", "python", "manage.py", "check"] test: ["CMD", "python3", "-c", "import http.client; c=http.client.HTTPConnection('localhost',8000,timeout=5); c.request('GET','/'); exit(0 if c.getresponse().status<400 else 1)"]
interval: 30s interval: 2m
timeout: 10s timeout: 10s
retries: 3 retries: 3
start_period: 40s
huey: huey:
build: . build: .

View file

@ -11,7 +11,6 @@ https://docs.djangoproject.com/en/5.2/ref/settings/
""" """
import os import os
import socket
from pathlib import Path from pathlib import Path
# Build paths inside the project like this: BASE_DIR / 'subdir'. # Build paths inside the project like this: BASE_DIR / 'subdir'.
@ -82,6 +81,9 @@ INSTALLED_APPS = [
"app.boosts.apps.BoostsConfig", "app.boosts.apps.BoostsConfig",
"app.interactions.apps.InteractionsConfig", "app.interactions.apps.InteractionsConfig",
"app.sse_notifications.apps.SseNotificationsConfig", "app.sse_notifications.apps.SseNotificationsConfig",
"app.profile.apps.ProfileConfig",
"app.stats.apps.StatsConfig",
"app.bridge.apps.BridgeConfig",
] ]
MIDDLEWARE = [ MIDDLEWARE = [
@ -152,23 +154,12 @@ REST_FRAMEWORK = {
} }
# Cache configuration # Cache configuration
# In DEBUG (dev/test) we use DummyCache so the suite runs without Redis.
# In production we always use Redis: a startup probe is unsafe because if Redis
# is still warming up at import time the process would silently fall back to
# DummyCache for its entire lifetime, breaking cache invalidation.
if DEBUG:
def redis_available():
"""Check if Redis is available"""
try:
redis_host = os.environ.get("REDIS_HOST", "redis")
redis_port = int(os.environ.get("REDIS_PORT", 6379))
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(1)
result = sock.connect_ex((redis_host, redis_port))
sock.close()
return result == 0
except Exception:
return False
if DEBUG or not redis_available():
CACHES = { CACHES = {
"default": { "default": {
"BACKEND": "django.core.cache.backends.dummy.DummyCache", "BACKEND": "django.core.cache.backends.dummy.DummyCache",

View file

@ -35,4 +35,7 @@ urlpatterns = [
path("polls/", include("app.polls.urls")), path("polls/", include("app.polls.urls")),
path("rss.xml", include("app.rss.urls")), path("rss.xml", include("app.rss.urls")),
path("sse/", include("app.sse_notifications.urls")), path("sse/", include("app.sse_notifications.urls")),
path("profile/", include("app.profile.urls")),
path("stats/", include("app.stats.urls")),
path("bridge/", include("app.bridge.urls")),
] ]

View file

@ -1,8 +1,7 @@
graph TD graph TD
List["📋 List nodes"] Node1["🖥️ Relay 1"]
Node1["🖥️ Node 1"] Node2["🖥️ Relay 2"]
Node2["🖥️ Node 2"] Node3["🖥️ Relay 3"]
Node3["🖥️ Node 3"]
%% Social.org instances with icons %% Social.org instances with icons
Social1_1["📄 social.org"] Social1_1["📄 social.org"]
@ -12,11 +11,6 @@ graph TD
Social3_1["📄 social.org"] Social3_1["📄 social.org"]
Social3_2["📄 social.org"] Social3_2["📄 social.org"]
%% Parent-child connections with labels
List -.->|"Get"| Node1
List -.->|"Get"| Node2
List -.->|"Get"| Node3
%% Node to social.org connections %% Node to social.org connections
Social1_1 -->|"⚓ Connects"| Node1 Social1_1 -->|"⚓ Connects"| Node1
Social1_2 -->|"⚓ Connects"| Node1 Social1_2 -->|"⚓ Connects"| Node1
@ -26,16 +20,14 @@ graph TD
Social3_2 -->|"⚓ Connects"| Node3 Social3_2 -->|"⚓ Connects"| Node3
%% Bidirectional connections between nodes %% Bidirectional connections between nodes
Node1 <-.->|"👥 Share Users"| Node2 Node1 <-.->|"👥 Share Feeds"| Node2
Node2 <-.->|"👥 Share Users"| Node3 Node2 <-.->|"👥 Share Feeds"| Node3
Node1 <-.->|"👥 Share Users"| Node3 Node1 <-.->|"👥 Share Feeds"| Node3
%% Modern color scheme with gradients %% Modern color scheme with gradients
classDef socialStyle fill:#667eea,stroke:#764ba2,stroke-width:3px,color:#fff,font-weight:bold classDef socialStyle fill:#667eea,stroke:#764ba2,stroke-width:3px,color:#fff,font-weight:bold
classDef nodeStyle fill:#f093fb,stroke:#f5576c,stroke-width:3px,color:#fff,font-weight:bold classDef nodeStyle fill:#f093fb,stroke:#f5576c,stroke-width:3px,color:#fff,font-weight:bold
classDef listStyle fill:#4facfe,stroke:#00f2fe,stroke-width:4px,color:#fff,font-weight:bold
%% Apply styles %% Apply styles
class Social1_1,Social1_2,Social2_1,Social2_2,Social3_1,Social3_2 socialStyle class Social1_1,Social1_2,Social2_1,Social2_2,Social3_1,Social3_2 socialStyle
class Node1,Node2,Node3 nodeStyle class Node1,Node2,Node3 nodeStyle
class List listStyle

View file

@ -11,6 +11,6 @@ python manage.py migrate
echo "🔍 Checking Django configuration..." echo "🔍 Checking Django configuration..."
python manage.py check python manage.py check
# Start Django development server # Start application server
echo "🎯 Starting Django development server on 0.0.0.0:8000..." echo "🎯 Starting uvicorn on 0.0.0.0:8000..."
exec python manage.py runserver 0.0.0.0:8000 exec uvicorn core.asgi:application --host 0.0.0.0 --port 8000 --workers 4

View file

@ -27,6 +27,15 @@ http {
proxy_send_timeout 60s; proxy_send_timeout 60s;
proxy_read_timeout 60s; proxy_read_timeout 60s;
# SSE location - disable buffering for streaming responses
location /sse/ {
proxy_pass http://django_app;
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 3600s;
gzip off;
}
# Main location # Main location
location / { location / {
proxy_pass http://django_app; proxy_pass http://django_app;

View file

@ -11,6 +11,8 @@ dependencies = [
"huey>=2.5.0", "huey>=2.5.0",
"redis>=4.0.0", "redis>=4.0.0",
"requests>=2.31.0", "requests>=2.31.0",
"python-dateutil>=2.8.0",
"feedparser>=6.0.0",
"pytest>=7.0.0", "pytest>=7.0.0",
"pytest-django>=4.5.0", "pytest-django>=4.5.0",
] ]

2
relay-list.txt Normal file
View file

@ -0,0 +1,2 @@
https://relay.org-social.org
https://orgs-relay.adsan.dev

View file

@ -6,6 +6,8 @@ redis>=4.0.0
django-redis>=5.0.0 django-redis>=5.0.0
requests>=2.31.0 requests>=2.31.0
python-dateutil>=2.8.0 python-dateutil>=2.8.0
feedparser>=6.0.0
pytest>=7.0.0 pytest>=7.0.0
pytest-django>=4.5.0 pytest-django>=4.5.0
org-python>=0.3.1 org-python>=0.3.1
uvicorn>=0.30.0