Compare commits
42 commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 3b1926198f | |||
| fdbf8461cd | |||
| 9c0e825fa1 | |||
| 2728620fd6 | |||
| c446ae514d | |||
| 349a5e1376 | |||
| 22271f2e0d | |||
| f7f85153f6 | |||
| 25b19b948f | |||
| 97ef297efe | |||
| 045e647ed2 | |||
| ee44472c74 | |||
| 0a66cfcf7a | |||
| 347369868d | |||
| 8b35862bb9 | |||
| 5a6e953786 | |||
|
|
109bb2f51c | ||
|
|
c7edc34f7f | ||
| a47aa3ad35 | |||
| 338a8b3121 | |||
| 5483f1b7a4 | |||
| 17fe6a9fca | |||
| 13d80cfa0f | |||
| 9a18e3e958 | |||
| 1d9a02c304 | |||
| 4bf0d8c1d0 | |||
| b13cf2fad4 | |||
| dc4813c5fc | |||
|
|
94d856a704 | ||
|
|
8ca15ffab9 | ||
|
|
9ccd326bea | ||
|
|
7c60f060c8 | ||
|
|
75460aa567 | ||
|
|
b8c3d98b5d | ||
|
|
0300a728d2 | ||
|
|
0873f18c00 | ||
|
|
aa5f200fb5 | ||
|
|
f09863d6a9 | ||
|
|
84ae9dffcc | ||
|
|
fd26468843 | ||
|
|
cc3a270d76 | ||
|
|
79ff3a63a5 |
95 changed files with 9724 additions and 268 deletions
7
CONTRIBUTING.md
Normal file
7
CONTRIBUTING.md
Normal file
|
|
@ -0,0 +1,7 @@
|
||||||
|
# Contributing
|
||||||
|
|
||||||
|
Thank you for your interest in contributing to this project!
|
||||||
|
|
||||||
|
For detailed contribution guidelines, please visit:
|
||||||
|
|
||||||
|
https://git.andros.dev/andros/contribute
|
||||||
|
|
@ -27,4 +27,4 @@ COPY . .
|
||||||
EXPOSE 8000
|
EXPOSE 8000
|
||||||
|
|
||||||
# Default command (can be overridden in docker-compose)
|
# Default command (can be overridden in docker-compose)
|
||||||
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
|
CMD ["uvicorn", "core.asgi:application", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]
|
||||||
581
README.md
581
README.md
|
|
@ -2,14 +2,13 @@
|
||||||
|
|
||||||
## Introduction
|
## Introduction
|
||||||
|
|
||||||
Org Social Relay is a P2P system that acts as an intermediary between all [Org Social](https://github.com/tanrax/org-social) files. It scans the network, creating an index of users, mentions, replies, groups and threads. This allows you to:
|
Org Social Relay is a P2P system that acts as an intermediary between all [Org Social](https://git.andros.dev/org-social/org-social) files. It scans the network, creating an index of users, mentions, replies, groups and threads. This allows you to:
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
graph TD
|
graph TD
|
||||||
List["📋 List nodes"]
|
Node1["🖥️ Relay"]
|
||||||
Node1["🖥️ Node 1"]
|
Node2["🖥️ Relay"]
|
||||||
Node2["🖥️ Node 2"]
|
Node3["🖥️ Relay"]
|
||||||
Node3["🖥️ Node 3"]
|
|
||||||
|
|
||||||
%% Social.org instances with icons
|
%% Social.org instances with icons
|
||||||
Social1_1["📄 social.org"]
|
Social1_1["📄 social.org"]
|
||||||
|
|
@ -19,11 +18,6 @@ graph TD
|
||||||
Social3_1["📄 social.org"]
|
Social3_1["📄 social.org"]
|
||||||
Social3_2["📄 social.org"]
|
Social3_2["📄 social.org"]
|
||||||
|
|
||||||
%% Parent-child connections with labels
|
|
||||||
List -.->|"Get"| Node1
|
|
||||||
List -.->|"Get"| Node2
|
|
||||||
List -.->|"Get"| Node3
|
|
||||||
|
|
||||||
%% Node to social.org connections
|
%% Node to social.org connections
|
||||||
Social1_1 -->|"⚓ Connects"| Node1
|
Social1_1 -->|"⚓ Connects"| Node1
|
||||||
Social1_2 -->|"⚓ Connects"| Node1
|
Social1_2 -->|"⚓ Connects"| Node1
|
||||||
|
|
@ -33,19 +27,17 @@ graph TD
|
||||||
Social3_2 -->|"⚓ Connects"| Node3
|
Social3_2 -->|"⚓ Connects"| Node3
|
||||||
|
|
||||||
%% Bidirectional connections between nodes
|
%% Bidirectional connections between nodes
|
||||||
Node1 <-.->|"👥 Share Users"| Node2
|
Node1 <-.->|"👥 Share Feeds"| Node2
|
||||||
Node2 <-.->|"👥 Share Users"| Node3
|
Node2 <-.->|"👥 Share Feeds"| Node3
|
||||||
Node1 <-.->|"👥 Share Users"| Node3
|
Node1 <-.->|"👥 Share Feeds"| Node3
|
||||||
|
|
||||||
%% Modern color scheme with gradients
|
%% Modern color scheme with gradients
|
||||||
classDef socialStyle fill:#667eea,stroke:#764ba2,stroke-width:3px,color:#fff,font-weight:bold
|
classDef socialStyle fill:#667eea,stroke:#764ba2,stroke-width:3px,color:#fff,font-weight:bold
|
||||||
classDef nodeStyle fill:#f093fb,stroke:#f5576c,stroke-width:3px,color:#fff,font-weight:bold
|
classDef nodeStyle fill:#f093fb,stroke:#f5576c,stroke-width:3px,color:#fff,font-weight:bold
|
||||||
classDef listStyle fill:#4facfe,stroke:#00f2fe,stroke-width:4px,color:#fff,font-weight:bold
|
|
||||||
|
|
||||||
%% Apply styles
|
%% Apply styles
|
||||||
class Social1_1,Social1_2,Social2_1,Social2_2,Social3_1,Social3_2 socialStyle
|
class Social1_1,Social1_2,Social2_1,Social2_2,Social3_1,Social3_2 socialStyle
|
||||||
class Node1,Node2,Node3 nodeStyle
|
class Node1,Node2,Node3 nodeStyle
|
||||||
class List listStyle
|
|
||||||
```
|
```
|
||||||
[Source](/diagram.mmd)
|
[Source](/diagram.mmd)
|
||||||
|
|
||||||
|
|
@ -54,12 +46,14 @@ graph TD
|
||||||
- Read or participate in threads.
|
- Read or participate in threads.
|
||||||
- Perform searches (tags and full text).
|
- Perform searches (tags and full text).
|
||||||
- Participate in groups.
|
- Participate in groups.
|
||||||
|
- See who boosted your posts.
|
||||||
|
- Notify external sites linked from posts via [Webmention](https://www.w3.org/TR/webmention/) (sending only; receiving them is not possible since plain text feeds cannot advertise an endpoint).
|
||||||
|
- Follow Mastodon/ActivityPub accounts and RSS/Atom feeds from any Org Social client through [bridges](#bridges-follow-activitypub-accounts-and-rss-feeds) (virtual social.org feeds).
|
||||||
|
|
||||||
## Concepts
|
## Concepts
|
||||||
|
|
||||||
- **List nodes**: Index of public nodes. Simple list with all the URLs of the Nodes (`https://cdn.jsdelivr.net/gh/tanrax/org-social/org-social-relay-list.txt`). It will be used by nodes to find other nodes and share information.
|
- **Relay**: A server running Org Social Relay (this software). Each relay maintains a local `relay-list.txt` file with URLs of other relays. Relays scan the network and share feed information with each other, creating a decentralized P2P network.
|
||||||
- **Node**: A server running Org Social Relay (this software). It scans the network and shares information with other nodes or clients.
|
- **Client**: An application that connects to a Relay to get information. It can be Org Social or any other application that implements the Org Social Relay API.
|
||||||
- **Client**: An application that connects to a Node to get information. It can be Org Social or any other application that implements the Org Social Relay API.
|
|
||||||
|
|
||||||
## Installation
|
## Installation
|
||||||
|
|
||||||
|
|
@ -92,13 +86,40 @@ nano .env
|
||||||
docker compose up -d
|
docker compose up -d
|
||||||
```
|
```
|
||||||
|
|
||||||
## Make your Org Social Relay public
|
## Managing Relay Nodes
|
||||||
|
|
||||||
If you want your Relay to be used by other users, and also communicate with other public Relays to work together scanning the network and improving everyone's speed, you must make a Pull Request to this file:
|
### Local Relay List
|
||||||
|
|
||||||
https://github.com/tanrax/org-social/blob/main/org-social-relay-list.txt
|
This relay uses a local file (`relay-list.txt` in the project root) to manage the list of other relay nodes it connects to.
|
||||||
|
|
||||||
Add your Relay URL (e.g. `https://my-relay.example.com`) in a new line.
|
To add or remove relay nodes from your local instance:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Edit the relay list file
|
||||||
|
nano relay-list.txt
|
||||||
|
|
||||||
|
# Add one relay URL per line, for example:
|
||||||
|
# https://relay.org-social.org
|
||||||
|
# https://other-relay.example.com
|
||||||
|
```
|
||||||
|
|
||||||
|
The relay will automatically discover feeds from these nodes every 3 hours.
|
||||||
|
|
||||||
|
### Make your Org Social Relay public
|
||||||
|
|
||||||
|
If you want your Relay to be discoverable by other relay nodes, you can make a Pull Request to add your relay URL to the `relay-list.txt` file in this repository:
|
||||||
|
|
||||||
|
**Steps to add your relay:**
|
||||||
|
|
||||||
|
1. Fork this repository
|
||||||
|
2. Edit `relay-list.txt` and add your relay URL on a new line:
|
||||||
|
```
|
||||||
|
https://your-relay.example.com
|
||||||
|
```
|
||||||
|
3. Create a Pull Request with your changes
|
||||||
|
4. Once merged, all relay nodes that update their code (via `git pull`) will automatically discover your relay
|
||||||
|
|
||||||
|
This way, the network of relays grows organically as new nodes are added to the shared list.
|
||||||
|
|
||||||
## Updating
|
## Updating
|
||||||
|
|
||||||
|
|
@ -153,6 +174,18 @@ curl -i http://localhost:8080/mentions/?feed=https://example.com/social.org
|
||||||
# Last-Modified: Wed, 01 Nov 2025 10:15:00 GMT
|
# Last-Modified: Wed, 01 Nov 2025 10:15:00 GMT
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### CORS (Cross-Origin Resource Sharing)
|
||||||
|
|
||||||
|
All Relay endpoints have CORS enabled with `Access-Control-Allow-Origin: *`, allowing direct access from any frontend/web application. This means you can call the API directly from JavaScript in the browser without CORS restrictions.
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```javascript
|
||||||
|
// Fetch notifications directly from the browser
|
||||||
|
fetch('http://localhost:8080/notifications/?feed=https://example.com/social.org')
|
||||||
|
.then(response => response.json())
|
||||||
|
.then(data => console.log(data));
|
||||||
|
```
|
||||||
|
|
||||||
### Root
|
### Root
|
||||||
|
|
||||||
`/` - Basic information about the relay.
|
`/` - Basic information about the relay.
|
||||||
|
|
@ -173,16 +206,26 @@ curl http://localhost:8080/
|
||||||
"self": {"href": "/", "method": "GET"},
|
"self": {"href": "/", "method": "GET"},
|
||||||
"feeds": {"href": "/feeds/", "method": "GET"},
|
"feeds": {"href": "/feeds/", "method": "GET"},
|
||||||
"add-feed": {"href": "/feeds/", "method": "POST"},
|
"add-feed": {"href": "/feeds/", "method": "POST"},
|
||||||
|
"feed-content": {"href": "/feed-content/?feed={feed_url}", "method": "GET", "templated": true},
|
||||||
"notifications": {"href": "/notifications/?feed={feed_url}", "method": "GET", "templated": true},
|
"notifications": {"href": "/notifications/?feed={feed_url}", "method": "GET", "templated": true},
|
||||||
|
"sse-notifications": {"href": "/sse/notifications/?feed={feed_url}", "method": "GET", "templated": true},
|
||||||
|
"sse-notifications-all": {"href": "/sse/notifications/", "method": "GET"},
|
||||||
"mentions": {"href": "/mentions/?feed={feed_url}", "method": "GET", "templated": true},
|
"mentions": {"href": "/mentions/?feed={feed_url}", "method": "GET", "templated": true},
|
||||||
"reactions": {"href": "/reactions/?feed={feed_url}", "method": "GET", "templated": true},
|
"reactions": {"href": "/reactions/?feed={feed_url}", "method": "GET", "templated": true},
|
||||||
"replies-to": {"href": "/replies-to/?feed={feed_url}", "method": "GET", "templated": true},
|
"replies-to": {"href": "/replies-to/?feed={feed_url}", "method": "GET", "templated": true},
|
||||||
|
"boosts": {"href": "/boosts/?post={post_url}", "method": "GET", "templated": true},
|
||||||
|
"interactions": {"href": "/interactions/?post={post_url}", "method": "GET", "templated": true},
|
||||||
"replies": {"href": "/replies/?post={post_url}", "method": "GET", "templated": true},
|
"replies": {"href": "/replies/?post={post_url}", "method": "GET", "templated": true},
|
||||||
"search": {"href": "/search/?q={query}", "method": "GET", "templated": true},
|
"search": {"href": "/search/?q={query}", "method": "GET", "templated": true},
|
||||||
"groups": {"href": "/groups/", "method": "GET"},
|
"groups": {"href": "/groups/", "method": "GET"},
|
||||||
"group-messages": {"href": "/groups/{group_slug}/", "method": "GET", "templated": true},
|
"group-messages": {"href": "/groups/{group_slug}/", "method": "GET", "templated": true},
|
||||||
"polls": {"href": "/polls/", "method": "GET"},
|
"polls": {"href": "/polls/", "method": "GET"},
|
||||||
"poll-votes": {"href": "/polls/votes/?post={post_url}", "method": "GET", "templated": true}
|
"poll-votes": {"href": "/polls/votes/?post={post_url}", "method": "GET", "templated": true},
|
||||||
|
"profile": {"href": "/profile/?feed={feed_url}", "method": "GET", "templated": true},
|
||||||
|
"rss": {"href": "/rss.xml", "method": "GET", "description": "RSS feed of latest posts (supports ?tag={tag} and ?feed={feed_url} filters)"},
|
||||||
|
"bridge": {"href": "/bridge/", "method": "GET"},
|
||||||
|
"bridge-activitypub": {"href": "/bridge/activitypub/@{user}@{instance}/", "method": "GET", "templated": true},
|
||||||
|
"bridge-rss": {"href": "/bridge/rss/?url={feed_url}", "method": "GET", "templated": true}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
@ -230,7 +273,7 @@ curl -X POST http://localhost:8080/feeds/ -d '{"feed": "https://example.com/path
|
||||||
|
|
||||||
### Get notifications
|
### Get notifications
|
||||||
|
|
||||||
`/notifications/?feed={url feed}` - Get all notifications (mentions, reactions, and replies) received by a given feed. Results are ordered from most recent to oldest.
|
`/notifications/?feed={url feed}` - Get all notifications (mentions, reactions, replies, and boosts) received by a given feed. Results are ordered from most recent to oldest.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
# URL must be encoded when passed as query parameter
|
# URL must be encoded when passed as query parameter
|
||||||
|
|
@ -245,6 +288,11 @@ curl -G "http://localhost:8080/notifications/" --data-urlencode "feed=https://ex
|
||||||
"type": "Success",
|
"type": "Success",
|
||||||
"errors": [],
|
"errors": [],
|
||||||
"data": [
|
"data": [
|
||||||
|
{
|
||||||
|
"type": "boost",
|
||||||
|
"post": "https://alice.org/social.org#2025-02-05T14:00:00+0100",
|
||||||
|
"boosted": "https://example.com/social.org#2025-02-05T10:00:00+0100"
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"type": "reaction",
|
"type": "reaction",
|
||||||
"post": "https://alice.org/social.org#2025-02-05T13:15:00+0100",
|
"post": "https://alice.org/social.org#2025-02-05T13:15:00+0100",
|
||||||
|
|
@ -269,11 +317,12 @@ curl -G "http://localhost:8080/notifications/" --data-urlencode "feed=https://ex
|
||||||
],
|
],
|
||||||
"meta": {
|
"meta": {
|
||||||
"feed": "https://example.com/social.org",
|
"feed": "https://example.com/social.org",
|
||||||
"total": 4,
|
"total": 5,
|
||||||
"by_type": {
|
"by_type": {
|
||||||
"mentions": 1,
|
"mentions": 1,
|
||||||
"reactions": 2,
|
"reactions": 2,
|
||||||
"replies": 1
|
"replies": 1,
|
||||||
|
"boosts": 1
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"_links": {
|
"_links": {
|
||||||
|
|
@ -286,22 +335,84 @@ curl -G "http://localhost:8080/notifications/" --data-urlencode "feed=https://ex
|
||||||
```
|
```
|
||||||
|
|
||||||
Each notification includes:
|
Each notification includes:
|
||||||
- `type`: The notification type (`"mention"`, `"reaction"`, or `"reply"`)
|
- `type`: The notification type (`"mention"`, `"reaction"`, `"reply"`, or `"boost"`)
|
||||||
- `post`: The notification post URL (format: `{author_feed}#{timestamp}`)
|
- `post`: The notification post URL (format: `{author_feed}#{timestamp}`)
|
||||||
- `emoji`: (Only for reactions) The reaction emoji
|
- `emoji`: (Only for reactions) The reaction emoji
|
||||||
- `parent`: (Only for reactions and replies) The post URL that received the notification
|
- `parent`: (Only for reactions and replies) The post URL that received the notification
|
||||||
|
- `boosted`: (Only for boosts) The original post URL that was boosted
|
||||||
|
|
||||||
**Mentions** only have `type` and `post` because you are the one being mentioned in someone else's post.
|
**Mentions** only have `type` and `post` because you are the one being mentioned in someone else's post.
|
||||||
|
|
||||||
**Reactions and replies** have `parent` to indicate which of your posts received the reaction/reply.
|
**Reactions and replies** have `parent` to indicate which of your posts received the reaction/reply.
|
||||||
|
|
||||||
|
**Boosts** have `boosted` to indicate which of your posts was shared.
|
||||||
|
|
||||||
To extract the author's feed from the `post` field, simply take the part before the `#` character. For example, from `https://alice.org/social.org#2025-02-05T13:15:00+0100`, the author is `https://alice.org/social.org`.
|
To extract the author's feed from the `post` field, simply take the part before the `#` character. For example, from `https://alice.org/social.org#2025-02-05T13:15:00+0100`, the author is `https://alice.org/social.org`.
|
||||||
|
|
||||||
The `by_type` breakdown in `meta` allows you to show notification counts per type in your UI.
|
The `by_type` breakdown in `meta` allows you to show notification counts per type in your UI.
|
||||||
|
|
||||||
**Optional parameters:**
|
**Optional parameters:**
|
||||||
- `type`: Filter by notification type (`mention`, `reaction`, `reply`)
|
- `type`: Filter by notification type (`mention`, `reaction`, `reply`, `boost`)
|
||||||
- Example: `/notifications/?feed={feed}&type=reaction`
|
- Example: `/notifications/?feed={feed}&type=reaction`
|
||||||
|
- Example: `/notifications/?feed={feed}&type=boost`
|
||||||
|
|
||||||
|
### Real-time notifications (SSE)
|
||||||
|
|
||||||
|
`/sse/notifications/` - Subscribe to real-time notifications via Server-Sent Events. The `?feed=` parameter is optional:
|
||||||
|
|
||||||
|
- **With `?feed=`**: streams only notifications for that feed.
|
||||||
|
- **Without `?feed=`**: streams all notifications from all feeds (intended for backends such as push notification services).
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# Per-feed stream
|
||||||
|
curl -N "http://localhost:8080/sse/notifications/?feed=https://example.com/social.org"
|
||||||
|
|
||||||
|
# Global stream (all feeds)
|
||||||
|
curl -N "http://localhost:8080/sse/notifications/"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Events:**
|
||||||
|
|
||||||
|
- `connected` - Connection established
|
||||||
|
```
|
||||||
|
event: connected
|
||||||
|
data: {"feed": "https://example.com/social.org", "status": "connected"}
|
||||||
|
```
|
||||||
|
In the global stream the `feed` field is omitted:
|
||||||
|
```
|
||||||
|
event: connected
|
||||||
|
data: {"status": "connected"}
|
||||||
|
```
|
||||||
|
|
||||||
|
- `heartbeat` - Connection keepalive (every 30s)
|
||||||
|
```
|
||||||
|
event: heartbeat
|
||||||
|
data: {"status": "alive", "timestamp": 1733392800}
|
||||||
|
```
|
||||||
|
|
||||||
|
- `notification` - New notification received (per-feed, same structure as `/notifications/`)
|
||||||
|
```
|
||||||
|
event: notification
|
||||||
|
data: {"type": "mention", "post": "https://alice.org/social.org#2025-02-05T11:20:00+0100"}
|
||||||
|
```
|
||||||
|
```
|
||||||
|
event: notification
|
||||||
|
data: {"type": "reply", "post": "https://bob.org/social.org#...", "parent": "https://example.com/social.org#..."}
|
||||||
|
```
|
||||||
|
```
|
||||||
|
event: notification
|
||||||
|
data: {"type": "reaction", "post": "https://carol.org/social.org#...", "emoji": "❤", "parent": "https://example.com/social.org#..."}
|
||||||
|
```
|
||||||
|
```
|
||||||
|
event: notification
|
||||||
|
data: {"type": "boost", "post": "https://dave.org/social.org#...", "boosted": "https://example.com/social.org#..."}
|
||||||
|
```
|
||||||
|
|
||||||
|
In the global stream, each notification includes a `target_feed` field identifying which feed received it:
|
||||||
|
```
|
||||||
|
event: notification
|
||||||
|
data: {"target_feed": "https://example.com/social.org", "type": "mention", "post": "https://alice.org/social.org#2025-02-05T11:20:00+0100"}
|
||||||
|
```
|
||||||
|
|
||||||
### Get mentions
|
### Get mentions
|
||||||
|
|
||||||
|
|
@ -437,6 +548,159 @@ To extract the author's feed from the `post` field, simply take the part before
|
||||||
|
|
||||||
**Note:** This endpoint shows direct replies to your posts. To see the full conversation thread of a specific post, use the `/replies/?post={post_url}` endpoint instead.
|
**Note:** This endpoint shows direct replies to your posts. To see the full conversation thread of a specific post, use the `/replies/?post={post_url}` endpoint instead.
|
||||||
|
|
||||||
|
### Get boosts
|
||||||
|
|
||||||
|
`/boosts/?post={url post}` - Get all boosts (reposts/shares) for a specific post. A boost is when someone shares your post on their timeline using the `:INCLUDE:` property. Results are ordered from most recent to oldest.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# URL must be encoded when passed as query parameter
|
||||||
|
curl "http://localhost:8080/boosts/?post=https%3A%2F%2Fexample.com%2Fsocial.org%232025-02-05T10%3A00%3A00%2B0100"
|
||||||
|
|
||||||
|
# Or use curl's --data-urlencode for automatic encoding:
|
||||||
|
curl -G "http://localhost:8080/boosts/" --data-urlencode "post=https://example.com/social.org#2025-02-05T10:00:00+0100"
|
||||||
|
```
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"type": "Success",
|
||||||
|
"errors": [],
|
||||||
|
"data": [
|
||||||
|
"https://alice.org/social.org#2025-02-05T14:00:00+0100",
|
||||||
|
"https://bob.org/social.org#2025-02-05T15:30:00+0100",
|
||||||
|
"https://charlie.org/social.org#2025-02-05T16:45:00+0100"
|
||||||
|
],
|
||||||
|
"meta": {
|
||||||
|
"post": "https://example.com/social.org#2025-02-05T10:00:00+0100",
|
||||||
|
"total": 3
|
||||||
|
},
|
||||||
|
"_links": {
|
||||||
|
"self": {"href": "/boosts/?post=https%3A%2F%2Fexample.com%2Fsocial.org%232025-02-05T10%3A00%3A00%2B0100", "method": "GET"}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The response includes:
|
||||||
|
- A list of boost post URLs (format: `{booster_feed}#{timestamp}`)
|
||||||
|
|
||||||
|
To extract the booster's feed from each post URL, simply take the part before the `#` character. For example, from `https://alice.org/social.org#2025-02-05T14:00:00+0100`, the booster is `https://alice.org/social.org`.
|
||||||
|
|
||||||
|
**Note:** According to Org Social specification, boosts are posts with the `:INCLUDE:` property pointing to the boosted post.
|
||||||
|
|
||||||
|
### Get interactions (all-in-one)
|
||||||
|
|
||||||
|
`/interactions/?post={url post}` - Get all interactions for a specific post in a single request. This endpoint consolidates reactions, replies, and boosts for optimal performance. Results are ordered from most recent to oldest.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# URL must be encoded when passed as query parameter
|
||||||
|
curl "http://localhost:8080/interactions/?post=https%3A%2F%2Fexample.com%2Fsocial.org%232025-02-05T10%3A00%3A00%2B0100"
|
||||||
|
|
||||||
|
# Or use curl's --data-urlencode for automatic encoding:
|
||||||
|
curl -G "http://localhost:8080/interactions/" --data-urlencode "post=https://example.com/social.org#2025-02-05T10:00:00+0100"
|
||||||
|
```
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"type": "Success",
|
||||||
|
"errors": [],
|
||||||
|
"data": {
|
||||||
|
"reactions": [
|
||||||
|
{
|
||||||
|
"post": "https://alice.org/social.org#2025-02-05T13:15:00+0100",
|
||||||
|
"emoji": "❤"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"post": "https://bob.org/social.org#2025-02-05T14:30:00+0100",
|
||||||
|
"emoji": "🚀"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"replies": [
|
||||||
|
"https://charlie.org/social.org#2025-02-05T12:30:00+0100",
|
||||||
|
"https://diana.org/social.org#2025-02-05T15:00:00+0100"
|
||||||
|
],
|
||||||
|
"boosts": [
|
||||||
|
"https://alice.org/social.org#2025-02-05T14:00:00+0100",
|
||||||
|
"https://bob.org/social.org#2025-02-05T15:30:00+0100"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"meta": {
|
||||||
|
"post": "https://example.com/social.org#2025-02-05T10:00:00+0100",
|
||||||
|
"total_reactions": 2,
|
||||||
|
"total_replies": 2,
|
||||||
|
"total_boosts": 2,
|
||||||
|
"parentChain": [
|
||||||
|
"https://original.org/social.org#2025-02-04T09:00:00+0100",
|
||||||
|
"https://parent.org/social.org#2025-02-05T08:00:00+0100"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"_links": {
|
||||||
|
"self": {"href": "/interactions/?post=https%3A%2F%2Fexample.com%2Fsocial.org%232025-02-05T10%3A00%3A00%2B0100", "method": "GET"},
|
||||||
|
"reactions": {"href": "/reactions/?feed=https%3A%2F%2Fexample.com%2Fsocial.org", "method": "GET"},
|
||||||
|
"replies": {"href": "/replies/?post=https%3A%2F%2Fexample.com%2Fsocial.org%232025-02-05T10%3A00%3A00%2B0100", "method": "GET"},
|
||||||
|
"boosts": {"href": "/boosts/?post=https%3A%2F%2Fexample.com%2Fsocial.org%232025-02-05T10%3A00%3A00%2B0100", "method": "GET"}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The response includes:
|
||||||
|
- `reactions`: Array of reaction objects with `post` (URL) and `emoji`
|
||||||
|
- `replies`: Array of reply post URLs (excludes reactions - posts without mood)
|
||||||
|
- `boosts`: Array of boost post URLs
|
||||||
|
- `parentChain`: Array of parent post URLs from oldest to most recent (empty if post is root)
|
||||||
|
|
||||||
|
**Note:** This endpoint is optimized for displaying post details in a single request. It excludes nested replies (use `/replies/?post={post_url}` for the full thread tree) and only includes direct replies to maintain simplicity and performance.
|
||||||
|
|
||||||
|
The `parentChain` allows you to reconstruct the conversation context by showing all parent posts from the original root post up to the immediate parent. If the post is a reply to another post, you'll get the full chain; if it's a root post, the array will be empty.
|
||||||
|
|
||||||
|
**Use cases:**
|
||||||
|
- Display a post with all its interactions in one request
|
||||||
|
- Show conversation context with parent chain
|
||||||
|
- Optimize mobile/web apps by reducing HTTP requests
|
||||||
|
- Get post engagement metrics (reactions, replies, boosts count)
|
||||||
|
|
||||||
|
### Get raw feed content
|
||||||
|
|
||||||
|
`/feed-content/?feed={url feed}` - Get the raw content of an Org Social feed file. This endpoint fetches and returns the original `.org` file content from the specified feed URL.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# URL must be encoded when passed as query parameter
|
||||||
|
curl "http://localhost:8080/feed-content/?feed=https%3A%2F%2Fexample.com%2Fsocial.org"
|
||||||
|
|
||||||
|
# Or use curl's --data-urlencode for automatic encoding:
|
||||||
|
curl -G "http://localhost:8080/feed-content/" --data-urlencode "feed=https://example.com/social.org"
|
||||||
|
```
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"type": "Success",
|
||||||
|
"errors": [],
|
||||||
|
"data": {
|
||||||
|
"content": "#+TITLE: My Social Feed\n#+AUTHOR: John Doe\n\n* 2025-02-05T10:00:00+0100\n:PROPERTIES:\n:ID: 2025-02-05T10:00:00+0100\n:END:\n\nHello, world! This is my first post.\n\n* 2025-02-05T12:30:00+0100\n:PROPERTIES:\n:ID: 2025-02-05T12:30:00+0100\n:REPLY_TO: https://alice.org/social.org#2025-02-05T10:15:00+0100\n:END:\n\nThis is a reply to Alice's post.\n"
|
||||||
|
},
|
||||||
|
"_links": {
|
||||||
|
"self": {"href": "/feed-content/?feed=https%3A%2F%2Fexample.com%2Fsocial.org", "method": "GET"}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The response includes:
|
||||||
|
- `content`: The raw text content of the feed file (Org Mode format)
|
||||||
|
|
||||||
|
**Use cases:**
|
||||||
|
- Debug feed format issues
|
||||||
|
- Parse feeds locally in client applications
|
||||||
|
- Backup or archive feed content
|
||||||
|
- Analyze feed structure and properties
|
||||||
|
- Validate Org Social format compliance
|
||||||
|
|
||||||
|
**Error handling:**
|
||||||
|
- Returns 400 if the `feed` parameter is missing or invalid
|
||||||
|
- Returns 404 if the feed is not registered in the relay
|
||||||
|
- Returns 502 if the feed URL cannot be fetched (network error, server down, etc.)
|
||||||
|
|
||||||
|
**Note:**
|
||||||
|
- This endpoint fetches the feed content directly from the source URL in real-time and does not use cached data.
|
||||||
|
- The content is returned exactly as stored in the original `.org` file, preserving all formatting, whitespace, and Org Mode properties.
|
||||||
|
|
||||||
### Get replies/threads
|
### Get replies/threads
|
||||||
|
|
||||||
`/replies/?post={url post}` - Get replies for a given post. This will return a tree structure with all the replies to posts in the given feed. If you want to see the entire tree, you must use the meta `parent` as a `post`.
|
`/replies/?post={url post}` - Get replies for a given post. This will return a tree structure with all the replies to posts in the given feed. If you want to see the entire tree, you must use the meta `parent` as a `post`.
|
||||||
|
|
@ -689,6 +953,48 @@ curl http://localhost:8080/polls/
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Get profile
|
||||||
|
|
||||||
|
`/profile/?feed={url feed}` - Get profile information for a given feed, including the list of feeds that follow it (followers).
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# URL must be encoded when passed as query parameter
|
||||||
|
curl "http://localhost:8080/profile/?feed=https%3A%2F%2Fexample.com%2Fsocial.org"
|
||||||
|
|
||||||
|
# Or use curl's --data-urlencode for automatic encoding:
|
||||||
|
curl -G "http://localhost:8080/profile/" --data-urlencode "feed=https://example.com/social.org"
|
||||||
|
```
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"type": "Success",
|
||||||
|
"errors": [],
|
||||||
|
"data": {
|
||||||
|
"feed": "https://example.com/social.org",
|
||||||
|
"followers": [
|
||||||
|
"https://alice.org/social.org",
|
||||||
|
"https://bob.org/social.org",
|
||||||
|
"https://charlie.org/social.org"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"meta": {
|
||||||
|
"feed": "https://example.com/social.org",
|
||||||
|
"total_followers": 3
|
||||||
|
},
|
||||||
|
"_links": {
|
||||||
|
"self": {"href": "/profile/?feed=https%3A%2F%2Fexample.com%2Fsocial.org", "method": "GET"}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The response includes:
|
||||||
|
- `feed`: The queried feed URL
|
||||||
|
- `followers`: List of feed URLs that follow this feed (i.e. feeds that have included this feed in their following list)
|
||||||
|
|
||||||
|
**Error handling:**
|
||||||
|
- Returns 400 if the `feed` parameter is missing or invalid
|
||||||
|
- Returns 404 if the feed is not registered in the relay
|
||||||
|
|
||||||
### Get poll votes
|
### Get poll votes
|
||||||
|
|
||||||
`/polls/votes/?post={url post}` - Get votes for a specific poll.
|
`/polls/votes/?post={url post}` - Get votes for a specific poll.
|
||||||
|
|
@ -741,6 +1047,183 @@ curl -G "http://localhost:8080/polls/votes/" --data-urlencode "post=https://foo.
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Get statistics
|
||||||
|
|
||||||
|
`/stats/` - Get aggregated statistics about the activity registered by the Relay. Data is grouped by year and month, plus a block of global counters. The month of each post is determined by its ID (RFC 3339 timestamp), not by when the Relay discovered it.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl "http://localhost:8080/stats/"
|
||||||
|
```
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"type": "Success",
|
||||||
|
"errors": [],
|
||||||
|
"data": {
|
||||||
|
"years": {
|
||||||
|
"2025": {
|
||||||
|
"11": {
|
||||||
|
"active_accounts": 42,
|
||||||
|
"total_posts": 512,
|
||||||
|
"posts": 300,
|
||||||
|
"replies": 120,
|
||||||
|
"boosts": 40,
|
||||||
|
"reactions": 35,
|
||||||
|
"group_messages": 15,
|
||||||
|
"polls": 7
|
||||||
|
},
|
||||||
|
"12": {
|
||||||
|
"active_accounts": 51,
|
||||||
|
"total_posts": 640,
|
||||||
|
"posts": 380,
|
||||||
|
"replies": 150,
|
||||||
|
"boosts": 52,
|
||||||
|
"reactions": 41,
|
||||||
|
"group_messages": 22,
|
||||||
|
"polls": 4
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"2026": {
|
||||||
|
"01": {
|
||||||
|
"active_accounts": 58,
|
||||||
|
"total_posts": 701,
|
||||||
|
"posts": 410,
|
||||||
|
"replies": 170,
|
||||||
|
"boosts": 60,
|
||||||
|
"reactions": 48,
|
||||||
|
"group_messages": 25,
|
||||||
|
"polls": 9
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"global": {
|
||||||
|
"registered_feeds": 128,
|
||||||
|
"total_accounts": 115,
|
||||||
|
"total_posts": 4210,
|
||||||
|
"total_follows": 340,
|
||||||
|
"active_groups": 6
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"meta": {
|
||||||
|
"generated_at": "2026-01-15T10:30:00+00:00"
|
||||||
|
},
|
||||||
|
"_links": {
|
||||||
|
"self": {"href": "/stats/", "method": "GET"},
|
||||||
|
"feeds": {"href": "/feeds/", "method": "GET"}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Counter definitions per month:
|
||||||
|
|
||||||
|
- `active_accounts`: Number of distinct accounts that published at least one post (of any kind) during that month.
|
||||||
|
- `total_posts`: Every post registered that month, regardless of its type.
|
||||||
|
- `posts`: Original posts, meaning posts that are neither a reply (`:REPLY_TO:`) nor a boost (`:INCLUDE:`).
|
||||||
|
- `replies`: Posts with the `:REPLY_TO:` property that contain text content. Reactions are excluded.
|
||||||
|
- `boosts`: Posts with the `:INCLUDE:` property.
|
||||||
|
- `reactions`: Replies with a `:MOOD:` property and no text content.
|
||||||
|
- `group_messages`: Posts published in any group (`:GROUP:` property).
|
||||||
|
- `polls`: Polls created that month (posts with `:POLL_END:`). Poll closings are not counted.
|
||||||
|
|
||||||
|
Global counters:
|
||||||
|
|
||||||
|
- `registered_feeds`: Total number of feeds currently registered in the Relay.
|
||||||
|
- `total_accounts`: Total number of accounts (profiles) known by the Relay.
|
||||||
|
- `total_posts`: Total number of posts registered, all time.
|
||||||
|
- `total_follows`: Total number of follow relationships between accounts.
|
||||||
|
- `active_groups`: Number of groups with at least one message.
|
||||||
|
|
||||||
|
**Note:** Counters are not mutually exclusive: a reply inside a group counts in both `replies` and `group_messages`, and a poll also counts in `posts`. Only `posts`, `replies`, `reactions` and `boosts` are disjoint between them, and together they add up to `total_posts`. Months with no activity are omitted. Like the rest of the endpoints, the response is cached and refreshed after each feed scan.
|
||||||
|
|
||||||
|
### Bridges (follow ActivityPub accounts and RSS feeds)
|
||||||
|
|
||||||
|
Bridges expose external accounts as **virtual social.org feeds** served by the relay. Any Org Social client can follow them with a plain `#+FOLLOW:` line, no client changes needed.
|
||||||
|
|
||||||
|
Registration is implicit: the first request of an unknown account fetches it from the origin and stores it. Active bridges are refreshed every 15 minutes; bridges nobody has requested for 90 days are removed automatically (and re-created on the next request).
|
||||||
|
|
||||||
|
#### ActivityPub bridge
|
||||||
|
|
||||||
|
`/bridge/activitypub/@{user}@{instance}/` - A Mastodon (or any ActivityPub) account as a virtual social.org feed (plain text).
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl http://localhost:8080/bridge/activitypub/@Mastodon@mastodon.social/
|
||||||
|
```
|
||||||
|
|
||||||
|
```org
|
||||||
|
#+TITLE: Mastodon
|
||||||
|
#+NICK: Mastodon
|
||||||
|
#+DESCRIPTION: Our mission is to connect the world...
|
||||||
|
#+AVATAR: https://files.mastodon.social/accounts/avatars/.../avatar.png
|
||||||
|
#+LINK: https://mastodon.social/@Mastodon
|
||||||
|
|
||||||
|
* Posts
|
||||||
|
** 2026-03-27T16:27:02+00:00
|
||||||
|
:PROPERTIES:
|
||||||
|
:LANG: en
|
||||||
|
:TAGS: fossback mastodon opensourcedesign ux
|
||||||
|
:END:
|
||||||
|
|
||||||
|
Following on from yesterday's blog post...
|
||||||
|
```
|
||||||
|
|
||||||
|
To follow the account, users add to their `social.org`:
|
||||||
|
|
||||||
|
```org
|
||||||
|
#+FOLLOW: https://relay.org-social.org/bridge/activitypub/@Mastodon@mastodon.social/
|
||||||
|
```
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
- Handles are case-insensitive; non-canonical spellings redirect (301) to the lowercase canonical URL.
|
||||||
|
- Only public top-level posts are bridged (no replies, no boosts), like Mastodon's public RSS.
|
||||||
|
- Content warnings become a `CW:` first line; media attachments become Org links; hashtags become `:TAGS:`.
|
||||||
|
- Instances running in *authorized fetch* mode (signed requests required) cannot be bridged.
|
||||||
|
|
||||||
|
`/bridge/activitypub/` (without handle) lists the accounts already bridged by this relay.
|
||||||
|
|
||||||
|
#### RSS/Atom bridge
|
||||||
|
|
||||||
|
`/bridge/rss/?url={feed_url}` - An RSS or Atom feed as a virtual social.org feed (plain text). The `url` parameter must be URL-encoded.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl -G http://localhost:8080/bridge/rss/ --data-urlencode "url=https://xkcd.com/rss.xml"
|
||||||
|
```
|
||||||
|
|
||||||
|
```org
|
||||||
|
#+TITLE: xkcd.com
|
||||||
|
#+NICK: xkcd_com
|
||||||
|
#+DESCRIPTION: xkcd.com: A webcomic of romance and math humor.
|
||||||
|
#+LINK: https://xkcd.com/
|
||||||
|
|
||||||
|
* Posts
|
||||||
|
** 2026-07-08T04:00:00+00:00
|
||||||
|
:PROPERTIES:
|
||||||
|
:END:
|
||||||
|
|
||||||
|
*** Airport Meeting
|
||||||
|
|
||||||
|
[[https://imgs.xkcd.com/comics/airport_meeting.png][Although it was a setback for physics...]]
|
||||||
|
|
||||||
|
[[https://xkcd.com/3269/]]
|
||||||
|
```
|
||||||
|
|
||||||
|
To follow the feed:
|
||||||
|
|
||||||
|
```org
|
||||||
|
#+FOLLOW: https://relay.org-social.org/bridge/rss/?url=https%3A%2F%2Fxkcd.com%2Frss.xml
|
||||||
|
```
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
- Entry titles become `***` sub-headings inside the post, followed by the converted content and a link to the original article.
|
||||||
|
- Entries without a publication date are skipped; entries sharing the same date get IDs shifted forward one second so none is lost.
|
||||||
|
|
||||||
|
`/bridge/rss/` (without `url`) lists the feeds already bridged by this relay.
|
||||||
|
|
||||||
|
#### Bridged feeds and the rest of the API
|
||||||
|
|
||||||
|
Bridged accounts are stored as regular profiles, so the existing endpoints (`/profile/`, `/search/`, `/rss.xml?feed=...`, ...) work with the bridge URL as the `feed` parameter. Bridged posts never trigger Webmentions nor notifications: the relay only mirrors external content.
|
||||||
|
|
||||||
## Groups Configuration
|
## Groups Configuration
|
||||||
|
|
||||||
Org Social Relay supports organizing posts into topic-based groups. Users can join groups to participate in focused discussions.
|
Org Social Relay supports organizing posts into topic-based groups. Users can join groups to participate in focused discussions.
|
||||||
|
|
@ -781,9 +1264,33 @@ Once configured, users can:
|
||||||
|
|
||||||
The groups endpoints will only be available when groups are configured via the `GROUPS` environment variable.
|
The groups endpoints will only be available when groups are configured via the `GROUPS` environment variable.
|
||||||
|
|
||||||
|
## RSS Feed
|
||||||
|
|
||||||
|
Org Social Relay provides an RSS feed of different types of content.
|
||||||
|
|
||||||
|
- The latest posts scanned from all registered feeds.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
http://localhost:8080/rss.xml
|
||||||
|
```
|
||||||
|
|
||||||
|
- By tag.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl http://localhost:8080/rss.xml?tag=emacs
|
||||||
|
```
|
||||||
|
|
||||||
|
- By author feed.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl http://localhost:8080/rss.xml?feed=https%3A%2F%2Forg-social.org%2Fsocial.org
|
||||||
|
```
|
||||||
|
|
||||||
|
This RSS feed can be used in RSS readers to stay updated with new posts from the relay, but is limited to the latest 200 posts.
|
||||||
|
|
||||||
## Technical information
|
## Technical information
|
||||||
|
|
||||||
You can find the public Relay list in `https://cdn.jsdelivr.net/gh/tanrax/org-social/org-social-relay-list.txt`.
|
The relay list is stored locally in `relay-list.txt`.
|
||||||
|
|
||||||
### Crons
|
### Crons
|
||||||
|
|
||||||
|
|
@ -804,3 +1311,19 @@ Every day at midnight, Relay analyzes the feeds of all registered users to disco
|
||||||
#### Cleanup stale feeds
|
#### Cleanup stale feeds
|
||||||
|
|
||||||
Every 3 days at 2 AM, Relay automatically removes feeds that haven't been successfully fetched (HTTP 200) in the last 3 days. This keeps the relay efficient by removing inactive or dead feeds.
|
Every 3 days at 2 AM, Relay automatically removes feeds that haven't been successfully fetched (HTTP 200) in the last 3 days. This keeps the relay efficient by removing inactive or dead feeds.
|
||||||
|
|
||||||
|
#### Refresh bridges
|
||||||
|
|
||||||
|
Every 15 minutes, Relay refetches the bridged ActivityPub accounts and RSS feeds that have been requested in the last 7 days. Dormant bridges are refreshed inline on their next request instead.
|
||||||
|
|
||||||
|
#### Cleanup stale bridges
|
||||||
|
|
||||||
|
Every 3 days at 3 AM, Relay removes bridged accounts that nobody has requested in the last 90 days, together with their stored profile and posts. They are re-created automatically if someone requests them again.
|
||||||
|
|
||||||
|
## Contributing
|
||||||
|
|
||||||
|
Contributions are welcome! Please see the [contribution guidelines](https://git.andros.dev/andros/contribute) for instructions on how to submit issues or pull requests.
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Org Social is free software, released under the [GPLv3 license](https://www.gnu.org/licenses/gpl-3.0.html).
|
||||||
|
|
|
||||||
0
app/boosts/__init__.py
Normal file
0
app/boosts/__init__.py
Normal file
6
app/boosts/apps.py
Normal file
6
app/boosts/apps.py
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
from django.apps import AppConfig
|
||||||
|
|
||||||
|
|
||||||
|
class BoostsConfig(AppConfig):
|
||||||
|
default_auto_field = "django.db.models.BigAutoField"
|
||||||
|
name = "app.boosts"
|
||||||
227
app/boosts/tests.py
Normal file
227
app/boosts/tests.py
Normal file
|
|
@ -0,0 +1,227 @@
|
||||||
|
from django.test import TestCase
|
||||||
|
from rest_framework.test import APIClient
|
||||||
|
from rest_framework import status
|
||||||
|
|
||||||
|
from app.feeds.models import Profile, Post
|
||||||
|
|
||||||
|
|
||||||
|
class BoostsViewTest(TestCase):
|
||||||
|
"""Test cases for the BoostsView API using Given/When/Then structure."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.client = APIClient()
|
||||||
|
self.boosts_url = "/boosts/"
|
||||||
|
|
||||||
|
# Create test profiles
|
||||||
|
self.profile1 = Profile.objects.create(
|
||||||
|
feed="https://example.com/social.org",
|
||||||
|
title="Example Profile",
|
||||||
|
nick="example_user",
|
||||||
|
description="Test profile 1",
|
||||||
|
)
|
||||||
|
self.profile2 = Profile.objects.create(
|
||||||
|
feed="https://alice.com/social.org",
|
||||||
|
title="Alice Profile",
|
||||||
|
nick="alice",
|
||||||
|
description="Alice's profile",
|
||||||
|
)
|
||||||
|
self.profile3 = Profile.objects.create(
|
||||||
|
feed="https://bob.com/social.org",
|
||||||
|
title="Bob Profile",
|
||||||
|
nick="bob",
|
||||||
|
description="Bob's profile",
|
||||||
|
)
|
||||||
|
self.profile4 = Profile.objects.create(
|
||||||
|
feed="https://charlie.com/social.org",
|
||||||
|
title="Charlie Profile",
|
||||||
|
nick="charlie",
|
||||||
|
description="Charlie's profile",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create original post
|
||||||
|
self.original_post = Post.objects.create(
|
||||||
|
profile=self.profile1,
|
||||||
|
post_id="2025-02-05T10:00:00+0100",
|
||||||
|
content="This is an amazing discovery!",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create boosts of the original post
|
||||||
|
self.boost1 = Post.objects.create(
|
||||||
|
profile=self.profile2,
|
||||||
|
post_id="2025-02-05T14:00:00+0100",
|
||||||
|
content="Guys, you have to see this!",
|
||||||
|
include=f"{self.profile1.feed}#{self.original_post.post_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.boost2 = Post.objects.create(
|
||||||
|
profile=self.profile3,
|
||||||
|
post_id="2025-02-05T15:30:00+0100",
|
||||||
|
content="", # Simple boost without comment
|
||||||
|
include=f"{self.profile1.feed}#{self.original_post.post_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.boost3 = Post.objects.create(
|
||||||
|
profile=self.profile4,
|
||||||
|
post_id="2025-02-05T16:45:00+0100",
|
||||||
|
content="", # Another simple boost
|
||||||
|
include=f"{self.profile1.feed}#{self.original_post.post_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create another post without boosts
|
||||||
|
self.post_without_boosts = Post.objects.create(
|
||||||
|
profile=self.profile1,
|
||||||
|
post_id="2025-02-05T11:00:00+0100",
|
||||||
|
content="Another post without boosts",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create a post that boosts a different post
|
||||||
|
self.other_post = Post.objects.create(
|
||||||
|
profile=self.profile2,
|
||||||
|
post_id="2025-02-05T09:00:00+0100",
|
||||||
|
content="Different post",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.boost_other = Post.objects.create(
|
||||||
|
profile=self.profile3,
|
||||||
|
post_id="2025-02-05T17:00:00+0100",
|
||||||
|
content="Boosting different post",
|
||||||
|
include=f"{self.profile2.feed}#{self.other_post.post_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_get_boosts_success(self):
|
||||||
|
"""Test GET /boosts/?post=<post_url> returns list of boosts."""
|
||||||
|
# Given: A post with boosts exists
|
||||||
|
post_url = f"{self.profile1.feed}#{self.original_post.post_id}"
|
||||||
|
|
||||||
|
# When: We request boosts for the post
|
||||||
|
response = self.client.get(self.boosts_url, {"post": post_url})
|
||||||
|
|
||||||
|
# Then: We should get boosts successfully
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
self.assertEqual(response.data["type"], "Success")
|
||||||
|
self.assertEqual(response.data["errors"], [])
|
||||||
|
|
||||||
|
# Then: Response should contain list of boost post URLs
|
||||||
|
data = response.data["data"]
|
||||||
|
self.assertIsInstance(data, list)
|
||||||
|
self.assertEqual(len(data), 3) # 3 boosts
|
||||||
|
|
||||||
|
# Then: Boosts should be ordered by post_id (most recent first)
|
||||||
|
expected_boosts = [
|
||||||
|
f"{self.profile4.feed}#{self.boost3.post_id}",
|
||||||
|
f"{self.profile3.feed}#{self.boost2.post_id}",
|
||||||
|
f"{self.profile2.feed}#{self.boost1.post_id}",
|
||||||
|
]
|
||||||
|
self.assertEqual(data, expected_boosts)
|
||||||
|
|
||||||
|
# Then: Meta should contain post URL and total count
|
||||||
|
meta = response.data["meta"]
|
||||||
|
self.assertEqual(meta["post"], post_url)
|
||||||
|
self.assertEqual(meta["total"], 3)
|
||||||
|
|
||||||
|
# Then: Links should contain self reference
|
||||||
|
links = response.data["_links"]
|
||||||
|
self.assertIn("self", links)
|
||||||
|
self.assertIn("/boosts/", links["self"]["href"])
|
||||||
|
|
||||||
|
def test_get_boosts_no_boosts(self):
|
||||||
|
"""Test GET /boosts/?post=<post_url> when post has no boosts."""
|
||||||
|
# Given: A post without boosts exists
|
||||||
|
post_url = f"{self.profile1.feed}#{self.post_without_boosts.post_id}"
|
||||||
|
|
||||||
|
# When: We request boosts for the post
|
||||||
|
response = self.client.get(self.boosts_url, {"post": post_url})
|
||||||
|
|
||||||
|
# Then: We should get successful response with empty list
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
self.assertEqual(response.data["type"], "Success")
|
||||||
|
data = response.data["data"]
|
||||||
|
self.assertEqual(len(data), 0)
|
||||||
|
self.assertEqual(response.data["meta"]["total"], 0)
|
||||||
|
|
||||||
|
def test_get_boosts_missing_post_parameter(self):
|
||||||
|
"""Test GET /boosts/ without post parameter returns error."""
|
||||||
|
# Given: No post parameter provided
|
||||||
|
|
||||||
|
# When: We request boosts without post parameter
|
||||||
|
response = self.client.get(self.boosts_url)
|
||||||
|
|
||||||
|
# Then: We should get a 400 error
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn("'post' parameter is required", response.data["errors"])
|
||||||
|
|
||||||
|
def test_get_boosts_invalid_post_url_format(self):
|
||||||
|
"""Test GET /boosts/?post=<invalid_url> returns error."""
|
||||||
|
# Given: An invalid post URL (missing #)
|
||||||
|
invalid_url = "https://example.com/social.org"
|
||||||
|
|
||||||
|
# When: We request boosts with invalid URL
|
||||||
|
response = self.client.get(self.boosts_url, {"post": invalid_url})
|
||||||
|
|
||||||
|
# Then: We should get a 400 error
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn("Invalid post URL format", response.data["errors"][0])
|
||||||
|
|
||||||
|
def test_get_boosts_post_not_found(self):
|
||||||
|
"""Test GET /boosts/?post=<nonexistent_post> returns 404."""
|
||||||
|
# Given: A non-existent post URL
|
||||||
|
post_url = "https://nonexistent.com/social.org#2025-01-01T00:00:00+00:00"
|
||||||
|
|
||||||
|
# When: We request boosts for non-existent post
|
||||||
|
response = self.client.get(self.boosts_url, {"post": post_url})
|
||||||
|
|
||||||
|
# Then: We should get a 404 error
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn("Post not found", response.data["errors"])
|
||||||
|
|
||||||
|
def test_get_boosts_profile_not_found(self):
|
||||||
|
"""Test GET /boosts/?post=<url_with_unknown_profile> returns 404."""
|
||||||
|
# Given: A URL with unknown profile
|
||||||
|
post_url = "https://unknown-profile.com/social.org#2025-01-01T00:00:00+00:00"
|
||||||
|
|
||||||
|
# When: We request boosts
|
||||||
|
response = self.client.get(self.boosts_url, {"post": post_url})
|
||||||
|
|
||||||
|
# Then: We should get a 404 error
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn("Post not found", response.data["errors"])
|
||||||
|
|
||||||
|
def test_boosts_isolated_by_post(self):
|
||||||
|
"""Test that boosts are correctly isolated by post."""
|
||||||
|
# Given: Multiple posts with boosts exist
|
||||||
|
post1_url = f"{self.profile1.feed}#{self.original_post.post_id}"
|
||||||
|
post2_url = f"{self.profile2.feed}#{self.other_post.post_id}"
|
||||||
|
|
||||||
|
# When: We request boosts for post 1
|
||||||
|
response1 = self.client.get(self.boosts_url, {"post": post1_url})
|
||||||
|
|
||||||
|
# Then: We should only get boosts for post 1
|
||||||
|
self.assertEqual(len(response1.data["data"]), 3)
|
||||||
|
|
||||||
|
# When: We request boosts for post 2
|
||||||
|
response2 = self.client.get(self.boosts_url, {"post": post2_url})
|
||||||
|
|
||||||
|
# Then: We should only get boosts for post 2
|
||||||
|
self.assertEqual(len(response2.data["data"]), 1)
|
||||||
|
self.assertIn(
|
||||||
|
f"{self.profile3.feed}#{self.boost_other.post_id}",
|
||||||
|
response2.data["data"],
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_boosts_caching(self):
|
||||||
|
"""Test that boosts responses are cached."""
|
||||||
|
# Given: A post with boosts
|
||||||
|
post_url = f"{self.profile1.feed}#{self.original_post.post_id}"
|
||||||
|
|
||||||
|
# When: We request boosts twice
|
||||||
|
response1 = self.client.get(self.boosts_url, {"post": post_url})
|
||||||
|
response2 = self.client.get(self.boosts_url, {"post": post_url})
|
||||||
|
|
||||||
|
# Then: Both responses should be identical
|
||||||
|
self.assertEqual(response1.data, response2.data)
|
||||||
|
self.assertEqual(response1.status_code, status.HTTP_200_OK)
|
||||||
|
self.assertEqual(response2.status_code, status.HTTP_200_OK)
|
||||||
8
app/boosts/urls.py
Normal file
8
app/boosts/urls.py
Normal file
|
|
@ -0,0 +1,8 @@
|
||||||
|
from django.urls import path
|
||||||
|
from .views import BoostsView
|
||||||
|
|
||||||
|
app_name = "boosts"
|
||||||
|
|
||||||
|
urlpatterns = [
|
||||||
|
path("", BoostsView.as_view(), name="boosts"),
|
||||||
|
]
|
||||||
22
app/boosts/utils.py
Normal file
22
app/boosts/utils.py
Normal file
|
|
@ -0,0 +1,22 @@
|
||||||
|
"""
|
||||||
|
Utility functions for boosts
|
||||||
|
"""
|
||||||
|
|
||||||
|
from app.feeds.models import Post
|
||||||
|
|
||||||
|
|
||||||
|
def get_boosts_for_post(post_url: str):
|
||||||
|
"""
|
||||||
|
Get all boosts for a specific post.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
post_url: The post URL in format feed_url#post_id
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
QuerySet of Post objects that boost the given post
|
||||||
|
"""
|
||||||
|
return (
|
||||||
|
Post.objects.filter(include=post_url)
|
||||||
|
.select_related("profile")
|
||||||
|
.order_by("-post_id")
|
||||||
|
)
|
||||||
97
app/boosts/views.py
Normal file
97
app/boosts/views.py
Normal file
|
|
@ -0,0 +1,97 @@
|
||||||
|
from rest_framework.views import APIView
|
||||||
|
from rest_framework.response import Response
|
||||||
|
from rest_framework import status
|
||||||
|
from django.core.cache import cache
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from app.feeds.models import Post, Profile
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class BoostsView(APIView):
|
||||||
|
"""Get boosts for a specific post"""
|
||||||
|
|
||||||
|
def get(self, request):
|
||||||
|
post_url = request.query_params.get("post")
|
||||||
|
|
||||||
|
if not post_url:
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Error",
|
||||||
|
"errors": ["'post' parameter is required"],
|
||||||
|
"data": None,
|
||||||
|
},
|
||||||
|
status=status.HTTP_400_BAD_REQUEST,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Parse the post URL format: https://feed.com/social.org#post_id
|
||||||
|
try:
|
||||||
|
if "#" not in post_url:
|
||||||
|
raise ValueError("Invalid post URL format")
|
||||||
|
feed_url, post_id = post_url.split("#", 1)
|
||||||
|
except ValueError:
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Error",
|
||||||
|
"errors": ["Invalid post URL format. Expected: feed_url#post_id"],
|
||||||
|
"data": None,
|
||||||
|
},
|
||||||
|
status=status.HTTP_400_BAD_REQUEST,
|
||||||
|
)
|
||||||
|
|
||||||
|
cache_key = f"boosts_{feed_url}_{post_id}"
|
||||||
|
cached_response = cache.get(cache_key)
|
||||||
|
|
||||||
|
if cached_response is not None:
|
||||||
|
return Response(cached_response, status=status.HTTP_200_OK)
|
||||||
|
|
||||||
|
# Find the original post
|
||||||
|
try:
|
||||||
|
profile = Profile.objects.get(feed=feed_url)
|
||||||
|
Post.objects.get(profile=profile, post_id=post_id)
|
||||||
|
except (Profile.DoesNotExist, Post.DoesNotExist):
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Error",
|
||||||
|
"errors": ["Post not found"],
|
||||||
|
"data": None,
|
||||||
|
},
|
||||||
|
status=status.HTTP_404_NOT_FOUND,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Get all boosts for this post
|
||||||
|
# A boost is a post with include property pointing to this post
|
||||||
|
original_post_url = f"{feed_url}#{post_id}"
|
||||||
|
|
||||||
|
boosts = (
|
||||||
|
Post.objects.filter(include=original_post_url)
|
||||||
|
.select_related("profile")
|
||||||
|
.order_by("-post_id")
|
||||||
|
)
|
||||||
|
|
||||||
|
# Build data according to README spec (simple list of post URLs)
|
||||||
|
boosts_data = [f"{boost.profile.feed}#{boost.post_id}" for boost in boosts]
|
||||||
|
|
||||||
|
# URL encode the post_url for the self link
|
||||||
|
from urllib.parse import quote
|
||||||
|
|
||||||
|
encoded_post_url = quote(post_url, safe="")
|
||||||
|
|
||||||
|
response_data = {
|
||||||
|
"type": "Success",
|
||||||
|
"errors": [],
|
||||||
|
"data": boosts_data,
|
||||||
|
"meta": {
|
||||||
|
"post": original_post_url,
|
||||||
|
"total": len(boosts_data),
|
||||||
|
},
|
||||||
|
"_links": {
|
||||||
|
"self": {"href": f"/boosts/?post={encoded_post_url}", "method": "GET"}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
# Cache permanently (will be cleared by scan_feeds task)
|
||||||
|
cache.set(cache_key, response_data, None)
|
||||||
|
|
||||||
|
return Response(response_data, status=status.HTTP_200_OK)
|
||||||
0
app/bridge/__init__.py
Normal file
0
app/bridge/__init__.py
Normal file
215
app/bridge/activitypub.py
Normal file
215
app/bridge/activitypub.py
Normal file
|
|
@ -0,0 +1,215 @@
|
||||||
|
"""
|
||||||
|
ActivityPub fetching for the bridge: WebFinger resolution, actor document
|
||||||
|
and outbox pagination, normalized into a bridge-agnostic structure.
|
||||||
|
|
||||||
|
Only public top-level notes are bridged: replies and boosts (Announce)
|
||||||
|
are skipped, mirroring what Mastodon exposes in its public RSS feeds.
|
||||||
|
Instances that require signed fetches (authorized fetch mode) will
|
||||||
|
answer 401/403 and the account cannot be bridged.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
from urllib.parse import quote
|
||||||
|
|
||||||
|
from dateutil import parser as date_parser
|
||||||
|
|
||||||
|
from .fetching import BridgeError, safe_get
|
||||||
|
from .html_to_org import html_to_org, html_to_text
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
HANDLE_RE = re.compile(r"^[a-z0-9._-]{1,64}@[a-z0-9-]+(\.[a-z0-9-]+)+$")
|
||||||
|
|
||||||
|
ACTIVITYPUB_ACCEPT = (
|
||||||
|
"application/activity+json, "
|
||||||
|
'application/ld+json; profile="https://www.w3.org/ns/activitystreams"'
|
||||||
|
)
|
||||||
|
MAX_POSTS = 40
|
||||||
|
MAX_OUTBOX_PAGES = 3
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_handle(raw_handle):
|
||||||
|
"""
|
||||||
|
Normalize a user@instance handle: lowercase and without the optional
|
||||||
|
leading @. Returns None if the result is not a valid handle.
|
||||||
|
"""
|
||||||
|
handle = (raw_handle or "").strip().lstrip("@").lower()
|
||||||
|
if not HANDLE_RE.match(handle):
|
||||||
|
return None
|
||||||
|
return handle
|
||||||
|
|
||||||
|
|
||||||
|
def _get_json(url, description):
|
||||||
|
try:
|
||||||
|
return json.loads(safe_get(url, accept=ACTIVITYPUB_ACCEPT))
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
raise BridgeError(f"Invalid JSON in {description} at {url}") from e
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_actor_url(user, instance):
|
||||||
|
"""Resolve the actor URL of user@instance through WebFinger."""
|
||||||
|
resource = quote(f"acct:{user}@{instance}", safe="")
|
||||||
|
webfinger_url = f"https://{instance}/.well-known/webfinger?resource={resource}"
|
||||||
|
try:
|
||||||
|
document = json.loads(safe_get(webfinger_url, accept="application/jrd+json"))
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
raise BridgeError(f"Invalid WebFinger response from {instance}") from e
|
||||||
|
|
||||||
|
for link in document.get("links", []):
|
||||||
|
if not isinstance(link, dict) or link.get("rel") != "self":
|
||||||
|
continue
|
||||||
|
if "activity+json" in (link.get("type") or "") and link.get("href"):
|
||||||
|
return link["href"]
|
||||||
|
for link in document.get("links", []):
|
||||||
|
if isinstance(link, dict) and link.get("rel") == "self" and link.get("href"):
|
||||||
|
return link["href"]
|
||||||
|
raise BridgeError(f"No ActivityPub actor found for {user}@{instance}")
|
||||||
|
|
||||||
|
|
||||||
|
def _format_post_id(published):
|
||||||
|
"""Convert an ActivityPub published date to an Org Social post ID."""
|
||||||
|
parsed = date_parser.parse(published)
|
||||||
|
if parsed.tzinfo is None:
|
||||||
|
return None
|
||||||
|
from datetime import timezone as dt_timezone
|
||||||
|
|
||||||
|
return parsed.astimezone(dt_timezone.utc).strftime("%Y-%m-%dT%H:%M:%S+00:00")
|
||||||
|
|
||||||
|
|
||||||
|
def _note_to_post(note):
|
||||||
|
"""Normalize an ActivityPub Note object into a bridge post dict."""
|
||||||
|
published = note.get("published")
|
||||||
|
if not published:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
post_id = _format_post_id(published)
|
||||||
|
except (ValueError, OverflowError):
|
||||||
|
logger.warning(f"Skipping note with unparseable date: {published}")
|
||||||
|
return None
|
||||||
|
if post_id is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
parts = []
|
||||||
|
summary = note.get("summary")
|
||||||
|
if summary:
|
||||||
|
parts.append(f"CW: {html_to_text(summary)}")
|
||||||
|
body = html_to_org(note.get("content") or "")
|
||||||
|
if body:
|
||||||
|
parts.append(body)
|
||||||
|
attachment_links = []
|
||||||
|
for attachment in note.get("attachment", []):
|
||||||
|
if isinstance(attachment, dict) and attachment.get("url"):
|
||||||
|
attachment_links.append(f"[[{attachment['url']}]]")
|
||||||
|
if attachment_links:
|
||||||
|
parts.append("\n".join(attachment_links))
|
||||||
|
|
||||||
|
tags = []
|
||||||
|
for tag in note.get("tag", []):
|
||||||
|
if isinstance(tag, dict) and tag.get("type") == "Hashtag" and tag.get("name"):
|
||||||
|
tags.append(tag["name"].lstrip("#"))
|
||||||
|
|
||||||
|
language = ""
|
||||||
|
content_map = note.get("contentMap")
|
||||||
|
if isinstance(content_map, dict) and content_map:
|
||||||
|
language = next(iter(content_map.keys()), "")
|
||||||
|
|
||||||
|
return {
|
||||||
|
"id": post_id,
|
||||||
|
"content": "\n\n".join(parts),
|
||||||
|
"tags": " ".join(tags),
|
||||||
|
"language": language,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _collect_outbox_posts(outbox_url):
|
||||||
|
"""Walk the outbox pages collecting public top-level notes."""
|
||||||
|
document = _get_json(outbox_url, "outbox")
|
||||||
|
|
||||||
|
if "orderedItems" in document:
|
||||||
|
page = document
|
||||||
|
else:
|
||||||
|
first = document.get("first")
|
||||||
|
if isinstance(first, str):
|
||||||
|
page = _get_json(first, "outbox page")
|
||||||
|
elif isinstance(first, dict):
|
||||||
|
page = first
|
||||||
|
else:
|
||||||
|
return []
|
||||||
|
|
||||||
|
posts = []
|
||||||
|
seen_ids = set()
|
||||||
|
pages_read = 0
|
||||||
|
|
||||||
|
while page and pages_read < MAX_OUTBOX_PAGES and len(posts) < MAX_POSTS:
|
||||||
|
pages_read += 1
|
||||||
|
for item in page.get("orderedItems", []):
|
||||||
|
if len(posts) >= MAX_POSTS:
|
||||||
|
break
|
||||||
|
if not isinstance(item, dict) or item.get("type") != "Create":
|
||||||
|
continue
|
||||||
|
note = item.get("object")
|
||||||
|
if not isinstance(note, dict) or note.get("type") != "Note":
|
||||||
|
continue
|
||||||
|
if note.get("inReplyTo"):
|
||||||
|
continue
|
||||||
|
post = _note_to_post(note)
|
||||||
|
if post is None or post["id"] in seen_ids:
|
||||||
|
continue
|
||||||
|
seen_ids.add(post["id"])
|
||||||
|
posts.append(post)
|
||||||
|
|
||||||
|
next_url = page.get("next")
|
||||||
|
if isinstance(next_url, str) and len(posts) < MAX_POSTS:
|
||||||
|
page = _get_json(next_url, "outbox page")
|
||||||
|
else:
|
||||||
|
page = None
|
||||||
|
|
||||||
|
return posts
|
||||||
|
|
||||||
|
|
||||||
|
def _actor_avatar(actor):
|
||||||
|
icon = actor.get("icon")
|
||||||
|
if isinstance(icon, list):
|
||||||
|
icon = icon[0] if icon else None
|
||||||
|
if isinstance(icon, dict):
|
||||||
|
return icon.get("url") or ""
|
||||||
|
if isinstance(icon, str):
|
||||||
|
return icon
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_account(user, instance):
|
||||||
|
"""
|
||||||
|
Fetch an ActivityPub account and normalize it for the bridge.
|
||||||
|
|
||||||
|
Returns a dict with actor_url, outbox_url, metadata and posts.
|
||||||
|
Raises BridgeError when the account cannot be fetched.
|
||||||
|
"""
|
||||||
|
actor_url = _resolve_actor_url(user, instance)
|
||||||
|
actor = _get_json(actor_url, "actor document")
|
||||||
|
|
||||||
|
outbox_url = actor.get("outbox")
|
||||||
|
nick = actor.get("preferredUsername") or user
|
||||||
|
metadata = {
|
||||||
|
"title": actor.get("name") or f"{nick}@{instance}",
|
||||||
|
"nick": nick,
|
||||||
|
"description": html_to_text(actor.get("summary") or ""),
|
||||||
|
"avatar": _actor_avatar(actor),
|
||||||
|
"link": actor.get("url") or actor_url,
|
||||||
|
}
|
||||||
|
|
||||||
|
posts = []
|
||||||
|
if isinstance(outbox_url, str) and outbox_url:
|
||||||
|
posts = _collect_outbox_posts(outbox_url)
|
||||||
|
else:
|
||||||
|
logger.warning(f"Actor {actor_url} has no outbox")
|
||||||
|
outbox_url = ""
|
||||||
|
|
||||||
|
return {
|
||||||
|
"actor_url": actor_url,
|
||||||
|
"outbox_url": outbox_url,
|
||||||
|
"metadata": metadata,
|
||||||
|
"posts": posts,
|
||||||
|
}
|
||||||
6
app/bridge/apps.py
Normal file
6
app/bridge/apps.py
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
from django.apps import AppConfig
|
||||||
|
|
||||||
|
|
||||||
|
class BridgeConfig(AppConfig):
|
||||||
|
default_auto_field = "django.db.models.BigAutoField"
|
||||||
|
name = "app.bridge"
|
||||||
52
app/bridge/fetching.py
Normal file
52
app/bridge/fetching.py
Normal file
|
|
@ -0,0 +1,52 @@
|
||||||
|
"""
|
||||||
|
Shared HTTP fetching helpers for the bridge app.
|
||||||
|
|
||||||
|
All bridge fetches go through safe_get(), which enforces the same SSRF
|
||||||
|
protections used for Webmentions, plus a timeout and a response size cap.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
|
||||||
|
import requests
|
||||||
|
|
||||||
|
from app.feeds.webmentions import is_safe_endpoint
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
BRIDGE_TIMEOUT = 10
|
||||||
|
BRIDGE_USER_AGENT = "Org-Social-Relay-Bridge (+https://relay.org-social.org/)"
|
||||||
|
# Maximum bytes read from any remote response
|
||||||
|
BRIDGE_MAX_BYTES = 5 * 1024 * 1024
|
||||||
|
|
||||||
|
|
||||||
|
class BridgeError(Exception):
|
||||||
|
"""A bridge source could not be fetched or understood."""
|
||||||
|
|
||||||
|
|
||||||
|
def safe_get(url, accept=None):
|
||||||
|
"""
|
||||||
|
GET a remote URL with SSRF protection, timeout and size cap.
|
||||||
|
|
||||||
|
Returns the raw bytes of the response body.
|
||||||
|
Raises BridgeError on unsafe URLs, network errors or non-2xx codes.
|
||||||
|
"""
|
||||||
|
if not is_safe_endpoint(url):
|
||||||
|
raise BridgeError(f"URL is not allowed: {url}")
|
||||||
|
|
||||||
|
headers = {"User-Agent": BRIDGE_USER_AGENT}
|
||||||
|
if accept:
|
||||||
|
headers["Accept"] = accept
|
||||||
|
|
||||||
|
try:
|
||||||
|
response = requests.get(
|
||||||
|
url, timeout=BRIDGE_TIMEOUT, headers=headers, stream=True
|
||||||
|
)
|
||||||
|
except requests.RequestException as e:
|
||||||
|
raise BridgeError(f"Could not reach {url}: {e}") from e
|
||||||
|
|
||||||
|
try:
|
||||||
|
if not (200 <= response.status_code < 300):
|
||||||
|
raise BridgeError(f"HTTP {response.status_code} from {url}")
|
||||||
|
return response.raw.read(BRIDGE_MAX_BYTES, decode_content=True)
|
||||||
|
finally:
|
||||||
|
response.close()
|
||||||
207
app/bridge/html_to_org.py
Normal file
207
app/bridge/html_to_org.py
Normal file
|
|
@ -0,0 +1,207 @@
|
||||||
|
"""
|
||||||
|
Minimal HTML to Org Mode text converter for bridged content.
|
||||||
|
|
||||||
|
Converts the HTML bodies published by ActivityPub servers and RSS feeds
|
||||||
|
into Org text suitable for a post body inside a virtual social.org file.
|
||||||
|
Only the common subset is handled (paragraphs, line breaks, links,
|
||||||
|
emphasis, lists, quotes and preformatted blocks); unknown tags are
|
||||||
|
stripped, keeping their text.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import re
|
||||||
|
from html.parser import HTMLParser
|
||||||
|
|
||||||
|
# A post body line must never look like an Org Social headline (* or **
|
||||||
|
# followed by whitespace) because it would break the virtual feed
|
||||||
|
# structure. Three or more asterisks are legal (post titles).
|
||||||
|
_HEADLINE_RE = re.compile(r"^(\*{1,2})(\s|$)")
|
||||||
|
|
||||||
|
_EMPHASIS_TAGS = {
|
||||||
|
"strong": "*",
|
||||||
|
"b": "*",
|
||||||
|
"em": "/",
|
||||||
|
"i": "/",
|
||||||
|
"code": "~",
|
||||||
|
"tt": "~",
|
||||||
|
}
|
||||||
|
|
||||||
|
_HEADING_TAGS = {"h1", "h2", "h3", "h4", "h5", "h6"}
|
||||||
|
|
||||||
|
|
||||||
|
def escape_org_lines(text):
|
||||||
|
"""
|
||||||
|
Prefix a space to any line that would be parsed as an Org Social
|
||||||
|
headline, so remote content cannot inject fake posts.
|
||||||
|
"""
|
||||||
|
lines = []
|
||||||
|
for line in text.split("\n"):
|
||||||
|
if _HEADLINE_RE.match(line):
|
||||||
|
line = " " + line
|
||||||
|
lines.append(line)
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
class _OrgHTMLConverter(HTMLParser):
|
||||||
|
def __init__(self):
|
||||||
|
super().__init__(convert_charrefs=True)
|
||||||
|
self.blocks = []
|
||||||
|
self.parts = []
|
||||||
|
self.anchor_stack = [] # (href, render_as_plain_text, position)
|
||||||
|
self.emphasis_stack = [] # (marker, position)
|
||||||
|
self.in_list_item = False
|
||||||
|
self.pre_depth = 0
|
||||||
|
self.pre_parts = []
|
||||||
|
self.invisible_depth = 0
|
||||||
|
|
||||||
|
# --- block helpers -------------------------------------------------
|
||||||
|
|
||||||
|
def _flush(self):
|
||||||
|
text = "".join(self.parts)
|
||||||
|
text = re.sub(r"[ \t]+", " ", text).strip()
|
||||||
|
self.parts = []
|
||||||
|
if not text:
|
||||||
|
return
|
||||||
|
if self.in_list_item:
|
||||||
|
self.blocks.append(f"- {text}")
|
||||||
|
else:
|
||||||
|
self.blocks.append(text)
|
||||||
|
|
||||||
|
def _append_block(self, block):
|
||||||
|
self._flush()
|
||||||
|
self.blocks.append(block)
|
||||||
|
|
||||||
|
# --- parser events -------------------------------------------------
|
||||||
|
|
||||||
|
def handle_starttag(self, tag, attrs):
|
||||||
|
if self.pre_depth:
|
||||||
|
if tag == "pre":
|
||||||
|
self.pre_depth += 1
|
||||||
|
return
|
||||||
|
|
||||||
|
attrs_dict = dict(attrs)
|
||||||
|
css_classes = (attrs_dict.get("class") or "").lower()
|
||||||
|
|
||||||
|
if tag in ("p", "div"):
|
||||||
|
self._flush()
|
||||||
|
elif tag == "br":
|
||||||
|
self.parts.append("\n")
|
||||||
|
elif tag == "a":
|
||||||
|
href = (attrs_dict.get("href") or "").strip()
|
||||||
|
plain = "mention" in css_classes or "hashtag" in css_classes
|
||||||
|
self.anchor_stack.append((href, plain, len(self.parts)))
|
||||||
|
elif tag in _EMPHASIS_TAGS:
|
||||||
|
self.emphasis_stack.append((_EMPHASIS_TAGS[tag], len(self.parts)))
|
||||||
|
elif tag in ("ul", "ol"):
|
||||||
|
self._flush()
|
||||||
|
elif tag == "li":
|
||||||
|
self._flush()
|
||||||
|
self.in_list_item = True
|
||||||
|
elif tag == "blockquote":
|
||||||
|
self._append_block("#+BEGIN_QUOTE")
|
||||||
|
elif tag == "pre":
|
||||||
|
self._flush()
|
||||||
|
self.pre_depth = 1
|
||||||
|
self.pre_parts = []
|
||||||
|
elif tag in _HEADING_TAGS:
|
||||||
|
self._flush()
|
||||||
|
self.emphasis_stack.append(("*", len(self.parts)))
|
||||||
|
elif tag == "img":
|
||||||
|
src = (attrs_dict.get("src") or "").strip()
|
||||||
|
if src and "]" not in src:
|
||||||
|
alt = " ".join((attrs_dict.get("alt") or "").split())
|
||||||
|
alt = alt.replace("]", ")")
|
||||||
|
if alt:
|
||||||
|
self.parts.append(f"[[{src}][{alt}]]")
|
||||||
|
else:
|
||||||
|
self.parts.append(f"[[{src}]]")
|
||||||
|
elif tag == "span" and "invisible" in css_classes:
|
||||||
|
self.invisible_depth += 1
|
||||||
|
|
||||||
|
def handle_endtag(self, tag):
|
||||||
|
if self.pre_depth:
|
||||||
|
if tag == "pre":
|
||||||
|
self.pre_depth -= 1
|
||||||
|
if self.pre_depth == 0:
|
||||||
|
code = "".join(self.pre_parts).strip("\n")
|
||||||
|
self._append_block(f"#+BEGIN_EXAMPLE\n{code}\n#+END_EXAMPLE")
|
||||||
|
return
|
||||||
|
|
||||||
|
if tag == "a" and self.anchor_stack:
|
||||||
|
href, plain, position = self.anchor_stack.pop()
|
||||||
|
text = "".join(self.parts[position:]).strip()
|
||||||
|
if plain or not href or "]" in href or "[[" in text:
|
||||||
|
# Org links cannot nest: anchors wrapping an image keep
|
||||||
|
# only the inner image link
|
||||||
|
rendered = text
|
||||||
|
elif not text or text == href:
|
||||||
|
rendered = f"[[{href}]]"
|
||||||
|
else:
|
||||||
|
text = text.replace("]", ")")
|
||||||
|
rendered = f"[[{href}][{text}]]"
|
||||||
|
self.parts[position:] = [rendered]
|
||||||
|
elif tag in _EMPHASIS_TAGS and self.emphasis_stack:
|
||||||
|
marker, position = self.emphasis_stack.pop()
|
||||||
|
text = "".join(self.parts[position:]).strip()
|
||||||
|
self.parts[position:] = [f"{marker}{text}{marker}"] if text else []
|
||||||
|
elif tag in ("p", "div", "ul", "ol"):
|
||||||
|
self._flush()
|
||||||
|
elif tag == "li":
|
||||||
|
self._flush()
|
||||||
|
self.in_list_item = False
|
||||||
|
elif tag == "blockquote":
|
||||||
|
self._append_block("#+END_QUOTE")
|
||||||
|
elif tag in _HEADING_TAGS and self.emphasis_stack:
|
||||||
|
marker, position = self.emphasis_stack.pop()
|
||||||
|
text = "".join(self.parts[position:]).strip()
|
||||||
|
self.parts[position:] = [f"{marker}{text}{marker}"] if text else []
|
||||||
|
self._flush()
|
||||||
|
elif tag == "span" and self.invisible_depth:
|
||||||
|
self.invisible_depth -= 1
|
||||||
|
|
||||||
|
def handle_data(self, data):
|
||||||
|
if self.pre_depth:
|
||||||
|
self.pre_parts.append(data)
|
||||||
|
return
|
||||||
|
if self.invisible_depth:
|
||||||
|
return
|
||||||
|
self.parts.append(data.replace("\n", " "))
|
||||||
|
|
||||||
|
def result(self):
|
||||||
|
self._flush()
|
||||||
|
# Consecutive list items stay together; everything else is
|
||||||
|
# separated by a blank line
|
||||||
|
chunks = []
|
||||||
|
previous_is_item = False
|
||||||
|
for block in self.blocks:
|
||||||
|
is_item = block.startswith("- ")
|
||||||
|
if chunks and is_item and previous_is_item:
|
||||||
|
chunks.append("\n")
|
||||||
|
elif chunks:
|
||||||
|
chunks.append("\n\n")
|
||||||
|
chunks.append(block)
|
||||||
|
previous_is_item = is_item
|
||||||
|
return "".join(chunks)
|
||||||
|
|
||||||
|
|
||||||
|
def html_to_org(html):
|
||||||
|
"""
|
||||||
|
Convert an HTML fragment to Org text. Returns plain text with Org
|
||||||
|
links/emphasis, paragraphs separated by blank lines, and headline
|
||||||
|
injection escaped. Never raises on malformed HTML.
|
||||||
|
"""
|
||||||
|
if not html:
|
||||||
|
return ""
|
||||||
|
converter = _OrgHTMLConverter()
|
||||||
|
try:
|
||||||
|
converter.feed(html)
|
||||||
|
converter.close()
|
||||||
|
except Exception:
|
||||||
|
# html.parser is very tolerant; this is a last-resort guard
|
||||||
|
return escape_org_lines(re.sub(r"<[^>]+>", " ", html).strip())
|
||||||
|
return escape_org_lines(converter.result())
|
||||||
|
|
||||||
|
|
||||||
|
def html_to_text(html):
|
||||||
|
"""Convert an HTML fragment to a single line of plain text."""
|
||||||
|
org = html_to_org(html)
|
||||||
|
return re.sub(r"\s+", " ", org).strip()
|
||||||
125
app/bridge/migrations/0001_initial.py
Normal file
125
app/bridge/migrations/0001_initial.py
Normal file
|
|
@ -0,0 +1,125 @@
|
||||||
|
# Generated by Django 6.0.7 on 2026-07-17 13:13
|
||||||
|
|
||||||
|
import django.db.models.deletion
|
||||||
|
from django.db import migrations, models
|
||||||
|
|
||||||
|
|
||||||
|
class Migration(migrations.Migration):
|
||||||
|
initial = True
|
||||||
|
|
||||||
|
dependencies = [
|
||||||
|
("feeds", "0011_outgoingwebmention"),
|
||||||
|
]
|
||||||
|
|
||||||
|
operations = [
|
||||||
|
migrations.CreateModel(
|
||||||
|
name="BridgedActivityPubAccount",
|
||||||
|
fields=[
|
||||||
|
(
|
||||||
|
"id",
|
||||||
|
models.BigAutoField(
|
||||||
|
auto_created=True,
|
||||||
|
primary_key=True,
|
||||||
|
serialize=False,
|
||||||
|
verbose_name="ID",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"handle",
|
||||||
|
models.CharField(
|
||||||
|
help_text="Normalized handle in user@instance form (lowercase)",
|
||||||
|
max_length=255,
|
||||||
|
unique=True,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"actor_url",
|
||||||
|
models.URLField(
|
||||||
|
help_text="ActivityPub actor document URL", max_length=500
|
||||||
|
),
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"outbox_url",
|
||||||
|
models.URLField(
|
||||||
|
blank=True, help_text="ActivityPub outbox URL", max_length=500
|
||||||
|
),
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"last_refreshed_at",
|
||||||
|
models.DateTimeField(
|
||||||
|
blank=True,
|
||||||
|
help_text="Last successful fetch from the origin",
|
||||||
|
null=True,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"last_accessed_at",
|
||||||
|
models.DateTimeField(
|
||||||
|
help_text="Last time a client requested this virtual feed"
|
||||||
|
),
|
||||||
|
),
|
||||||
|
("created_at", models.DateTimeField(auto_now_add=True)),
|
||||||
|
(
|
||||||
|
"profile",
|
||||||
|
models.OneToOneField(
|
||||||
|
help_text="Profile that stores the bridged data",
|
||||||
|
on_delete=django.db.models.deletion.CASCADE,
|
||||||
|
related_name="activitypub_bridge",
|
||||||
|
to="feeds.profile",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
],
|
||||||
|
options={
|
||||||
|
"ordering": ["handle"],
|
||||||
|
},
|
||||||
|
),
|
||||||
|
migrations.CreateModel(
|
||||||
|
name="BridgedRssFeed",
|
||||||
|
fields=[
|
||||||
|
(
|
||||||
|
"id",
|
||||||
|
models.BigAutoField(
|
||||||
|
auto_created=True,
|
||||||
|
primary_key=True,
|
||||||
|
serialize=False,
|
||||||
|
verbose_name="ID",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"source_url",
|
||||||
|
models.URLField(
|
||||||
|
help_text="URL of the RSS/Atom feed",
|
||||||
|
max_length=500,
|
||||||
|
unique=True,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"last_refreshed_at",
|
||||||
|
models.DateTimeField(
|
||||||
|
blank=True,
|
||||||
|
help_text="Last successful fetch from the origin",
|
||||||
|
null=True,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"last_accessed_at",
|
||||||
|
models.DateTimeField(
|
||||||
|
help_text="Last time a client requested this virtual feed"
|
||||||
|
),
|
||||||
|
),
|
||||||
|
("created_at", models.DateTimeField(auto_now_add=True)),
|
||||||
|
(
|
||||||
|
"profile",
|
||||||
|
models.OneToOneField(
|
||||||
|
help_text="Profile that stores the bridged data",
|
||||||
|
on_delete=django.db.models.deletion.CASCADE,
|
||||||
|
related_name="rss_bridge",
|
||||||
|
to="feeds.profile",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
],
|
||||||
|
options={
|
||||||
|
"ordering": ["source_url"],
|
||||||
|
},
|
||||||
|
),
|
||||||
|
]
|
||||||
0
app/bridge/migrations/__init__.py
Normal file
0
app/bridge/migrations/__init__.py
Normal file
125
app/bridge/models.py
Normal file
125
app/bridge/models.py
Normal file
|
|
@ -0,0 +1,125 @@
|
||||||
|
from urllib.parse import quote, urlparse
|
||||||
|
|
||||||
|
from django.conf import settings
|
||||||
|
from django.db import models
|
||||||
|
|
||||||
|
# Path fragments that identify a bridge virtual feed, on this relay or any other
|
||||||
|
BRIDGE_PATH_MARKERS = ("/bridge/activitypub/", "/bridge/rss/")
|
||||||
|
|
||||||
|
|
||||||
|
def is_bridge_feed_url(url):
|
||||||
|
"""
|
||||||
|
True when the URL points to a bridge virtual feed.
|
||||||
|
|
||||||
|
Bridges help users connect to external content but are not real
|
||||||
|
Org Social accounts, so they must stay out of the feed registry
|
||||||
|
and the statistics.
|
||||||
|
"""
|
||||||
|
path = urlparse(url).path
|
||||||
|
return any(marker in path for marker in BRIDGE_PATH_MARKERS)
|
||||||
|
|
||||||
|
|
||||||
|
def bridge_urls_q(field):
|
||||||
|
"""
|
||||||
|
Q object matching rows whose `field` URL belongs to a bridge virtual
|
||||||
|
feed. Use with exclude() to leave bridges out of listings and stats.
|
||||||
|
"""
|
||||||
|
q = models.Q()
|
||||||
|
for marker in BRIDGE_PATH_MARKERS:
|
||||||
|
q |= models.Q(**{f"{field}__contains": marker})
|
||||||
|
return q
|
||||||
|
|
||||||
|
|
||||||
|
def bridge_base_url():
|
||||||
|
"""
|
||||||
|
Base URL of this relay, used to build the public URL of virtual feeds.
|
||||||
|
"""
|
||||||
|
domain = settings.SITE_DOMAIN
|
||||||
|
scheme = "http" if domain.startswith(("localhost", "127.")) else "https"
|
||||||
|
return f"{scheme}://{domain}"
|
||||||
|
|
||||||
|
|
||||||
|
def activitypub_feed_url(handle):
|
||||||
|
"""Public URL of the virtual feed of a bridged ActivityPub account."""
|
||||||
|
return f"{bridge_base_url()}/bridge/activitypub/@{handle}/"
|
||||||
|
|
||||||
|
|
||||||
|
def rss_feed_url(source_url):
|
||||||
|
"""Public URL of the virtual feed of a bridged RSS feed."""
|
||||||
|
return f"{bridge_base_url()}/bridge/rss/?url={quote(source_url, safe='')}"
|
||||||
|
|
||||||
|
|
||||||
|
class BridgedActivityPubAccount(models.Model):
|
||||||
|
"""
|
||||||
|
An ActivityPub account (e.g. a Mastodon user) exposed as a virtual
|
||||||
|
Org Social feed at /bridge/activitypub/@{handle}/.
|
||||||
|
"""
|
||||||
|
|
||||||
|
handle = models.CharField(
|
||||||
|
max_length=255,
|
||||||
|
unique=True,
|
||||||
|
help_text="Normalized handle in user@instance form (lowercase)",
|
||||||
|
)
|
||||||
|
actor_url = models.URLField(
|
||||||
|
max_length=500, help_text="ActivityPub actor document URL"
|
||||||
|
)
|
||||||
|
outbox_url = models.URLField(
|
||||||
|
max_length=500, blank=True, help_text="ActivityPub outbox URL"
|
||||||
|
)
|
||||||
|
profile = models.OneToOneField(
|
||||||
|
"feeds.Profile",
|
||||||
|
on_delete=models.CASCADE,
|
||||||
|
related_name="activitypub_bridge",
|
||||||
|
help_text="Profile that stores the bridged data",
|
||||||
|
)
|
||||||
|
last_refreshed_at = models.DateTimeField(
|
||||||
|
null=True, blank=True, help_text="Last successful fetch from the origin"
|
||||||
|
)
|
||||||
|
last_accessed_at = models.DateTimeField(
|
||||||
|
help_text="Last time a client requested this virtual feed"
|
||||||
|
)
|
||||||
|
created_at = models.DateTimeField(auto_now_add=True)
|
||||||
|
|
||||||
|
class Meta:
|
||||||
|
ordering = ["handle"]
|
||||||
|
|
||||||
|
def __str__(self):
|
||||||
|
return f"@{self.handle}"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def feed_url(self):
|
||||||
|
return activitypub_feed_url(self.handle)
|
||||||
|
|
||||||
|
|
||||||
|
class BridgedRssFeed(models.Model):
|
||||||
|
"""
|
||||||
|
An RSS/Atom feed exposed as a virtual Org Social feed at
|
||||||
|
/bridge/rss/?url={source_url}.
|
||||||
|
"""
|
||||||
|
|
||||||
|
source_url = models.URLField(
|
||||||
|
max_length=500, unique=True, help_text="URL of the RSS/Atom feed"
|
||||||
|
)
|
||||||
|
profile = models.OneToOneField(
|
||||||
|
"feeds.Profile",
|
||||||
|
on_delete=models.CASCADE,
|
||||||
|
related_name="rss_bridge",
|
||||||
|
help_text="Profile that stores the bridged data",
|
||||||
|
)
|
||||||
|
last_refreshed_at = models.DateTimeField(
|
||||||
|
null=True, blank=True, help_text="Last successful fetch from the origin"
|
||||||
|
)
|
||||||
|
last_accessed_at = models.DateTimeField(
|
||||||
|
help_text="Last time a client requested this virtual feed"
|
||||||
|
)
|
||||||
|
created_at = models.DateTimeField(auto_now_add=True)
|
||||||
|
|
||||||
|
class Meta:
|
||||||
|
ordering = ["source_url"]
|
||||||
|
|
||||||
|
def __str__(self):
|
||||||
|
return self.source_url
|
||||||
|
|
||||||
|
@property
|
||||||
|
def feed_url(self):
|
||||||
|
return rss_feed_url(self.source_url)
|
||||||
54
app/bridge/org_renderer.py
Normal file
54
app/bridge/org_renderer.py
Normal file
|
|
@ -0,0 +1,54 @@
|
||||||
|
"""
|
||||||
|
Renders a bridged Profile and its Posts as a virtual social.org file.
|
||||||
|
|
||||||
|
The output follows the Org Social syntax so that any client can consume
|
||||||
|
the bridge URL with a plain #+FOLLOW: line.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .html_to_org import escape_org_lines
|
||||||
|
|
||||||
|
|
||||||
|
def _meta_value(value):
|
||||||
|
"""Metadata values must stay on a single line."""
|
||||||
|
return " ".join(str(value).split())
|
||||||
|
|
||||||
|
|
||||||
|
def render_profile_org(profile, posts):
|
||||||
|
"""
|
||||||
|
Build the virtual social.org content for a bridged profile.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
profile: app.feeds.models.Profile instance
|
||||||
|
posts: iterable of app.feeds.models.Post, oldest first
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
str: the Org Social file content
|
||||||
|
"""
|
||||||
|
lines = []
|
||||||
|
lines.append(f"#+TITLE: {_meta_value(profile.title) or profile.nick}")
|
||||||
|
lines.append(f"#+NICK: {_meta_value(profile.nick).replace(' ', '_')}")
|
||||||
|
if profile.description:
|
||||||
|
lines.append(f"#+DESCRIPTION: {_meta_value(profile.description)}")
|
||||||
|
if profile.avatar:
|
||||||
|
lines.append(f"#+AVATAR: {_meta_value(profile.avatar)}")
|
||||||
|
for link in profile.links.all():
|
||||||
|
lines.append(f"#+LINK: {_meta_value(link.url)}")
|
||||||
|
|
||||||
|
lines.append("")
|
||||||
|
lines.append("* Posts")
|
||||||
|
|
||||||
|
for post in posts:
|
||||||
|
lines.append(f"** {post.post_id}")
|
||||||
|
lines.append(":PROPERTIES:")
|
||||||
|
if post.language:
|
||||||
|
lines.append(f":LANG: {_meta_value(post.language)}")
|
||||||
|
if post.tags:
|
||||||
|
lines.append(f":TAGS: {_meta_value(post.tags)}")
|
||||||
|
lines.append(":END:")
|
||||||
|
lines.append("")
|
||||||
|
content = escape_org_lines(post.content.strip())
|
||||||
|
if content:
|
||||||
|
lines.append(content)
|
||||||
|
lines.append("")
|
||||||
|
|
||||||
|
return "\n".join(lines) + "\n"
|
||||||
139
app/bridge/rss.py
Normal file
139
app/bridge/rss.py
Normal file
|
|
@ -0,0 +1,139 @@
|
||||||
|
"""
|
||||||
|
RSS/Atom fetching for the bridge, normalized into the same structure
|
||||||
|
used by the ActivityPub bridge.
|
||||||
|
|
||||||
|
Each entry becomes a post whose body is the entry title as an Org
|
||||||
|
sub-heading (***), the converted entry content, and a link to the
|
||||||
|
original article.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import calendar
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
from datetime import datetime, timezone as dt_timezone
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
import feedparser
|
||||||
|
|
||||||
|
from .fetching import BridgeError, safe_get
|
||||||
|
from .html_to_org import html_to_org
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
MAX_POSTS = 40
|
||||||
|
|
||||||
|
_NICK_ALLOWED_RE = re.compile(r"[^A-Za-z0-9_-]+")
|
||||||
|
|
||||||
|
|
||||||
|
def _derive_nick(title, source_url):
|
||||||
|
"""Build a spaces-free nick from the feed title or the feed host."""
|
||||||
|
candidate = _NICK_ALLOWED_RE.sub("_", (title or "").strip()).strip("_")
|
||||||
|
if candidate:
|
||||||
|
return candidate
|
||||||
|
host = urlparse(source_url).hostname or "feed"
|
||||||
|
return host.replace(".", "_")
|
||||||
|
|
||||||
|
|
||||||
|
def _entry_post_id(entry):
|
||||||
|
"""Org Social post ID (UTC RFC 3339) from the entry date, or None."""
|
||||||
|
parsed_time = entry.get("published_parsed") or entry.get("updated_parsed")
|
||||||
|
if not parsed_time:
|
||||||
|
return None
|
||||||
|
timestamp = calendar.timegm(parsed_time)
|
||||||
|
return datetime.fromtimestamp(timestamp, dt_timezone.utc).strftime(
|
||||||
|
"%Y-%m-%dT%H:%M:%S+00:00"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _next_free_id(post_id, seen_ids):
|
||||||
|
"""
|
||||||
|
Entries sharing the same timestamp (e.g. date-only feeds) get IDs
|
||||||
|
shifted forward one second at a time so no post is lost.
|
||||||
|
"""
|
||||||
|
while post_id in seen_ids:
|
||||||
|
parsed = datetime.strptime(post_id, "%Y-%m-%dT%H:%M:%S+00:00")
|
||||||
|
shifted = parsed.timestamp() + 1
|
||||||
|
post_id = datetime.fromtimestamp(shifted, dt_timezone.utc).strftime(
|
||||||
|
"%Y-%m-%dT%H:%M:%S+00:00"
|
||||||
|
)
|
||||||
|
return post_id
|
||||||
|
|
||||||
|
|
||||||
|
def _entry_body(entry):
|
||||||
|
contents = entry.get("content") or []
|
||||||
|
if contents and contents[0].get("value"):
|
||||||
|
html = contents[0]["value"]
|
||||||
|
else:
|
||||||
|
html = entry.get("summary") or ""
|
||||||
|
return html_to_org(html)
|
||||||
|
|
||||||
|
|
||||||
|
def _entry_to_post(entry, seen_ids):
|
||||||
|
post_id = _entry_post_id(entry)
|
||||||
|
if post_id is None:
|
||||||
|
logger.warning(
|
||||||
|
f"Skipping RSS entry without date: {entry.get('link', '(no link)')}"
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
post_id = _next_free_id(post_id, seen_ids)
|
||||||
|
|
||||||
|
parts = []
|
||||||
|
title = " ".join((entry.get("title") or "").split())
|
||||||
|
if title:
|
||||||
|
parts.append(f"*** {title}")
|
||||||
|
body = _entry_body(entry)
|
||||||
|
if body:
|
||||||
|
parts.append(body)
|
||||||
|
link = (entry.get("link") or "").strip()
|
||||||
|
if link and "]" not in link:
|
||||||
|
parts.append(f"[[{link}]]")
|
||||||
|
|
||||||
|
tags = " ".join(
|
||||||
|
tag.get("term", "").strip().replace(" ", "-")
|
||||||
|
for tag in entry.get("tags", [])
|
||||||
|
if tag.get("term")
|
||||||
|
)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"id": post_id,
|
||||||
|
"content": "\n\n".join(parts),
|
||||||
|
"tags": tags,
|
||||||
|
"language": "",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_rss_feed(source_url):
|
||||||
|
"""
|
||||||
|
Fetch an RSS/Atom feed and normalize it for the bridge.
|
||||||
|
|
||||||
|
Returns a dict with metadata and posts.
|
||||||
|
Raises BridgeError when the feed cannot be fetched or parsed.
|
||||||
|
"""
|
||||||
|
raw = safe_get(source_url)
|
||||||
|
document = feedparser.parse(raw)
|
||||||
|
|
||||||
|
feed_info = document.get("feed", {})
|
||||||
|
if not document.get("entries") and not feed_info.get("title"):
|
||||||
|
raise BridgeError(f"Not a valid RSS/Atom feed: {source_url}")
|
||||||
|
|
||||||
|
title = " ".join((feed_info.get("title") or "").split())
|
||||||
|
image = feed_info.get("image") or {}
|
||||||
|
metadata = {
|
||||||
|
"title": title or source_url,
|
||||||
|
"nick": _derive_nick(title, source_url),
|
||||||
|
"description": " ".join((feed_info.get("subtitle") or "").split()),
|
||||||
|
"avatar": image.get("href", ""),
|
||||||
|
"link": feed_info.get("link") or source_url,
|
||||||
|
"language": (feed_info.get("language") or "").split("-")[0],
|
||||||
|
}
|
||||||
|
|
||||||
|
posts = []
|
||||||
|
seen_ids = set()
|
||||||
|
for entry in document.get("entries", [])[:MAX_POSTS]:
|
||||||
|
post = _entry_to_post(entry, seen_ids)
|
||||||
|
if post is None:
|
||||||
|
continue
|
||||||
|
seen_ids.add(post["id"])
|
||||||
|
posts.append(post)
|
||||||
|
|
||||||
|
return {"metadata": metadata, "posts": posts}
|
||||||
185
app/bridge/store.py
Normal file
185
app/bridge/store.py
Normal file
|
|
@ -0,0 +1,185 @@
|
||||||
|
"""
|
||||||
|
Persistence and orchestration for bridged accounts.
|
||||||
|
|
||||||
|
Normalized bridge data (from activitypub.py or rss.py) is stored in the
|
||||||
|
same Profile/Post tables used for real Org Social feeds, so every
|
||||||
|
existing relay endpoint (/profile/, /search/, /rss.xml, ...) works on
|
||||||
|
bridged accounts transparently.
|
||||||
|
|
||||||
|
Bridged posts never queue Webmentions nor publish notifications: the
|
||||||
|
content belongs to external authors, the relay only mirrors it.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from dateutil import parser as date_parser
|
||||||
|
from django.utils import timezone
|
||||||
|
|
||||||
|
from app.feeds.models import Post, Profile, ProfileLink
|
||||||
|
|
||||||
|
from . import activitypub, rss
|
||||||
|
from .models import (
|
||||||
|
BridgedActivityPubAccount,
|
||||||
|
BridgedRssFeed,
|
||||||
|
activitypub_feed_url,
|
||||||
|
rss_feed_url,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def store_bridge_data(feed_url, data):
|
||||||
|
"""
|
||||||
|
Create or update the Profile and Posts of a bridged account.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
feed_url: public URL of the virtual feed on this relay
|
||||||
|
data: normalized dict with "metadata" and "posts"
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Profile: the stored profile
|
||||||
|
"""
|
||||||
|
metadata = data["metadata"]
|
||||||
|
posts_data = data["posts"]
|
||||||
|
|
||||||
|
content_hash = hashlib.md5(f"{metadata}{posts_data}".encode()).hexdigest()
|
||||||
|
|
||||||
|
nick = " ".join((metadata.get("nick") or "").split()).replace(" ", "_")
|
||||||
|
profile_fields = {
|
||||||
|
"title": metadata.get("title", ""),
|
||||||
|
"nick": nick,
|
||||||
|
"description": metadata.get("description", ""),
|
||||||
|
"avatar": metadata.get("avatar", ""),
|
||||||
|
"language": metadata.get("language", ""),
|
||||||
|
"version": content_hash,
|
||||||
|
}
|
||||||
|
|
||||||
|
profile, created = Profile.objects.get_or_create(
|
||||||
|
feed=feed_url, defaults=profile_fields
|
||||||
|
)
|
||||||
|
|
||||||
|
if not created:
|
||||||
|
if profile.version == content_hash:
|
||||||
|
return profile
|
||||||
|
for field, value in profile_fields.items():
|
||||||
|
setattr(profile, field, value)
|
||||||
|
profile.save()
|
||||||
|
|
||||||
|
profile.links.all().delete()
|
||||||
|
link = (metadata.get("link") or "").strip()
|
||||||
|
if link:
|
||||||
|
ProfileLink.objects.create(profile=profile, url=link)
|
||||||
|
|
||||||
|
fetched_ids = set()
|
||||||
|
oldest_created_at = None
|
||||||
|
for post_data in posts_data:
|
||||||
|
post_id = post_data["id"]
|
||||||
|
try:
|
||||||
|
post_created_at = date_parser.parse(post_id)
|
||||||
|
except (ValueError, OverflowError):
|
||||||
|
logger.warning(f"Skipping bridged post with invalid ID: {post_id}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
fetched_ids.add(post_id)
|
||||||
|
if oldest_created_at is None or post_created_at < oldest_created_at:
|
||||||
|
oldest_created_at = post_created_at
|
||||||
|
|
||||||
|
post, post_created = Post.objects.get_or_create(
|
||||||
|
profile=profile,
|
||||||
|
post_id=post_id,
|
||||||
|
defaults={
|
||||||
|
"content": post_data["content"],
|
||||||
|
"tags": post_data.get("tags", "")[:500],
|
||||||
|
"language": post_data.get("language", "")[:10],
|
||||||
|
"created_at": post_created_at,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
if not post_created:
|
||||||
|
post.content = post_data["content"]
|
||||||
|
post.tags = post_data.get("tags", "")[:500]
|
||||||
|
post.language = post_data.get("language", "")[:10]
|
||||||
|
post.save()
|
||||||
|
|
||||||
|
# Posts deleted at the origin disappear from the fetched window;
|
||||||
|
# remove them, but never touch posts older than what was fetched
|
||||||
|
if fetched_ids and oldest_created_at is not None:
|
||||||
|
Post.objects.filter(profile=profile, created_at__gte=oldest_created_at).exclude(
|
||||||
|
post_id__in=fetched_ids
|
||||||
|
).delete()
|
||||||
|
|
||||||
|
return profile
|
||||||
|
|
||||||
|
|
||||||
|
def create_activitypub_bridge(handle):
|
||||||
|
"""
|
||||||
|
Fetch an ActivityPub account for the first time and create its bridge.
|
||||||
|
|
||||||
|
Raises BridgeError when the account cannot be fetched.
|
||||||
|
"""
|
||||||
|
user, instance = handle.split("@", 1)
|
||||||
|
data = activitypub.fetch_account(user, instance)
|
||||||
|
|
||||||
|
profile = store_bridge_data(activitypub_feed_url(handle), data)
|
||||||
|
|
||||||
|
now = timezone.now()
|
||||||
|
bridge = BridgedActivityPubAccount.objects.create(
|
||||||
|
handle=handle,
|
||||||
|
actor_url=data["actor_url"],
|
||||||
|
outbox_url=data["outbox_url"],
|
||||||
|
profile=profile,
|
||||||
|
last_refreshed_at=now,
|
||||||
|
last_accessed_at=now,
|
||||||
|
)
|
||||||
|
logger.info(f"Created ActivityPub bridge for @{handle}")
|
||||||
|
return bridge
|
||||||
|
|
||||||
|
|
||||||
|
def refresh_activitypub_bridge(bridge):
|
||||||
|
"""
|
||||||
|
Refetch a bridged ActivityPub account and update its stored data.
|
||||||
|
|
||||||
|
Raises BridgeError when the origin cannot be fetched.
|
||||||
|
"""
|
||||||
|
user, instance = bridge.handle.split("@", 1)
|
||||||
|
data = activitypub.fetch_account(user, instance)
|
||||||
|
store_bridge_data(bridge.feed_url, data)
|
||||||
|
|
||||||
|
bridge.actor_url = data["actor_url"] or bridge.actor_url
|
||||||
|
bridge.outbox_url = data["outbox_url"] or bridge.outbox_url
|
||||||
|
bridge.last_refreshed_at = timezone.now()
|
||||||
|
bridge.save()
|
||||||
|
|
||||||
|
|
||||||
|
def create_rss_bridge(source_url):
|
||||||
|
"""
|
||||||
|
Fetch an RSS/Atom feed for the first time and create its bridge.
|
||||||
|
|
||||||
|
Raises BridgeError when the feed cannot be fetched.
|
||||||
|
"""
|
||||||
|
data = rss.fetch_rss_feed(source_url)
|
||||||
|
|
||||||
|
profile = store_bridge_data(rss_feed_url(source_url), data)
|
||||||
|
|
||||||
|
now = timezone.now()
|
||||||
|
bridge = BridgedRssFeed.objects.create(
|
||||||
|
source_url=source_url,
|
||||||
|
profile=profile,
|
||||||
|
last_refreshed_at=now,
|
||||||
|
last_accessed_at=now,
|
||||||
|
)
|
||||||
|
logger.info(f"Created RSS bridge for {source_url}")
|
||||||
|
return bridge
|
||||||
|
|
||||||
|
|
||||||
|
def refresh_rss_bridge(bridge):
|
||||||
|
"""
|
||||||
|
Refetch a bridged RSS feed and update its stored data.
|
||||||
|
|
||||||
|
Raises BridgeError when the origin cannot be fetched.
|
||||||
|
"""
|
||||||
|
data = rss.fetch_rss_feed(bridge.source_url)
|
||||||
|
store_bridge_data(bridge.feed_url, data)
|
||||||
|
|
||||||
|
bridge.last_refreshed_at = timezone.now()
|
||||||
|
bridge.save()
|
||||||
110
app/bridge/tasks.py
Normal file
110
app/bridge/tasks.py
Normal file
|
|
@ -0,0 +1,110 @@
|
||||||
|
"""
|
||||||
|
Periodic maintenance of bridged accounts.
|
||||||
|
|
||||||
|
Only bridges requested recently are refreshed; dormant ones are
|
||||||
|
reactivated by the inline refresh on their next GET, and bridges nobody
|
||||||
|
has requested for a long time are deleted together with their profiles.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from datetime import timedelta
|
||||||
|
|
||||||
|
from huey import crontab
|
||||||
|
from huey.contrib.djhuey import periodic_task
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# Bridges accessed within this window are refreshed periodically
|
||||||
|
BRIDGE_ACTIVE_DAYS = 7
|
||||||
|
# Bridges not accessed for this long are deleted
|
||||||
|
BRIDGE_STALE_DAYS = 90
|
||||||
|
|
||||||
|
|
||||||
|
def _refresh_bridges_impl():
|
||||||
|
"""
|
||||||
|
Refresh every bridge accessed within the active window.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
dict: Counters of refreshed / failed bridges
|
||||||
|
"""
|
||||||
|
from django.utils import timezone
|
||||||
|
|
||||||
|
from .fetching import BridgeError
|
||||||
|
from .models import BridgedActivityPubAccount, BridgedRssFeed
|
||||||
|
from .store import refresh_activitypub_bridge, refresh_rss_bridge
|
||||||
|
|
||||||
|
cutoff = timezone.now() - timedelta(days=BRIDGE_ACTIVE_DAYS)
|
||||||
|
counters = {"refreshed": 0, "failed": 0}
|
||||||
|
|
||||||
|
refresh_plan = [
|
||||||
|
(BridgedActivityPubAccount, refresh_activitypub_bridge),
|
||||||
|
(BridgedRssFeed, refresh_rss_bridge),
|
||||||
|
]
|
||||||
|
|
||||||
|
for model, refresh in refresh_plan:
|
||||||
|
for bridge in model.objects.filter(last_accessed_at__gte=cutoff):
|
||||||
|
try:
|
||||||
|
refresh(bridge)
|
||||||
|
counters["refreshed"] += 1
|
||||||
|
except BridgeError as e:
|
||||||
|
counters["failed"] += 1
|
||||||
|
logger.warning(f"Failed to refresh bridge {bridge}: {e}")
|
||||||
|
except Exception as e:
|
||||||
|
counters["failed"] += 1
|
||||||
|
logger.error(f"Unexpected error refreshing bridge {bridge}: {e}")
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
f"Bridge refresh completed. "
|
||||||
|
f"Refreshed: {counters['refreshed']}, Failed: {counters['failed']}"
|
||||||
|
)
|
||||||
|
return counters
|
||||||
|
|
||||||
|
|
||||||
|
@periodic_task(crontab(minute="*/15")) # Run every 15 minutes
|
||||||
|
def refresh_bridges():
|
||||||
|
"""Periodic task to keep active bridged accounts up to date."""
|
||||||
|
import django
|
||||||
|
|
||||||
|
django.setup()
|
||||||
|
|
||||||
|
return _refresh_bridges_impl()
|
||||||
|
|
||||||
|
|
||||||
|
def _cleanup_stale_bridges_impl():
|
||||||
|
"""
|
||||||
|
Delete bridges (and their profiles with all posts) that nobody has
|
||||||
|
requested in BRIDGE_STALE_DAYS days.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
int: Number of bridges deleted
|
||||||
|
"""
|
||||||
|
from django.utils import timezone
|
||||||
|
|
||||||
|
from app.feeds.models import Profile
|
||||||
|
|
||||||
|
cutoff = timezone.now() - timedelta(days=BRIDGE_STALE_DAYS)
|
||||||
|
|
||||||
|
# Deleting the Profile cascades to the bridge row and its posts
|
||||||
|
stale_profiles = Profile.objects.filter(
|
||||||
|
activitypub_bridge__last_accessed_at__lt=cutoff
|
||||||
|
) | Profile.objects.filter(rss_bridge__last_accessed_at__lt=cutoff)
|
||||||
|
|
||||||
|
deleted = 0
|
||||||
|
for profile in stale_profiles:
|
||||||
|
logger.info(f"Deleting stale bridge profile: {profile.feed}")
|
||||||
|
profile.delete()
|
||||||
|
deleted += 1
|
||||||
|
|
||||||
|
if deleted:
|
||||||
|
logger.info(f"Stale bridge cleanup completed. Deleted {deleted} bridges")
|
||||||
|
return deleted
|
||||||
|
|
||||||
|
|
||||||
|
@periodic_task(crontab(day="*/3", hour=3, minute=0)) # Every 3 days at 3 AM
|
||||||
|
def cleanup_stale_bridges():
|
||||||
|
"""Periodic task to delete bridges nobody requests anymore."""
|
||||||
|
import django
|
||||||
|
|
||||||
|
django.setup()
|
||||||
|
|
||||||
|
return _cleanup_stale_bridges_impl()
|
||||||
295
app/bridge/test_activitypub.py
Normal file
295
app/bridge/test_activitypub.py
Normal file
|
|
@ -0,0 +1,295 @@
|
||||||
|
import json
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from django.test import SimpleTestCase
|
||||||
|
|
||||||
|
from app.bridge.activitypub import (
|
||||||
|
MAX_POSTS,
|
||||||
|
fetch_account,
|
||||||
|
normalize_handle,
|
||||||
|
)
|
||||||
|
from app.bridge.fetching import BridgeError
|
||||||
|
|
||||||
|
WEBFINGER_URL = (
|
||||||
|
"https://mastodon.example/.well-known/webfinger"
|
||||||
|
"?resource=acct%3Aalice%40mastodon.example"
|
||||||
|
)
|
||||||
|
ACTOR_URL = "https://mastodon.example/users/alice"
|
||||||
|
OUTBOX_URL = "https://mastodon.example/users/alice/outbox"
|
||||||
|
OUTBOX_PAGE_URL = "https://mastodon.example/users/alice/outbox?page=true"
|
||||||
|
|
||||||
|
|
||||||
|
def _note_item(published, content, **extra):
|
||||||
|
note = {"type": "Note", "published": published, "content": content}
|
||||||
|
note.update(extra)
|
||||||
|
return {"type": "Create", "object": note}
|
||||||
|
|
||||||
|
|
||||||
|
def _default_responses():
|
||||||
|
return {
|
||||||
|
WEBFINGER_URL: {
|
||||||
|
"links": [
|
||||||
|
{"rel": "http://webfinger.net/rel/profile-page"},
|
||||||
|
{
|
||||||
|
"rel": "self",
|
||||||
|
"type": "application/activity+json",
|
||||||
|
"href": ACTOR_URL,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
},
|
||||||
|
ACTOR_URL: {
|
||||||
|
"preferredUsername": "alice",
|
||||||
|
"name": "Alice",
|
||||||
|
"summary": "<p>My <strong>bio</strong></p>",
|
||||||
|
"icon": {"url": "https://mastodon.example/avatar.png"},
|
||||||
|
"url": "https://mastodon.example/@alice",
|
||||||
|
"outbox": OUTBOX_URL,
|
||||||
|
},
|
||||||
|
OUTBOX_URL: {"first": OUTBOX_PAGE_URL},
|
||||||
|
OUTBOX_PAGE_URL: {
|
||||||
|
"orderedItems": [
|
||||||
|
_note_item(
|
||||||
|
"2025-05-02T10:00:00Z",
|
||||||
|
"<p>Second post</p>",
|
||||||
|
tag=[{"type": "Hashtag", "name": "#emacs"}],
|
||||||
|
contentMap={"es": "<p>Segundo post</p>"},
|
||||||
|
),
|
||||||
|
_note_item(
|
||||||
|
"2025-05-01T18:00:00Z",
|
||||||
|
"<p>Reply post</p>",
|
||||||
|
inReplyTo="https://other.example/notes/1",
|
||||||
|
),
|
||||||
|
{"type": "Announce", "object": "https://other.example/notes/2"},
|
||||||
|
_note_item(
|
||||||
|
"2025-05-01T10:00:00Z",
|
||||||
|
"<p>First post</p>",
|
||||||
|
summary="Spoilers inside",
|
||||||
|
attachment=[
|
||||||
|
{
|
||||||
|
"type": "Document",
|
||||||
|
"url": "https://mastodon.example/media/cat.png",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
),
|
||||||
|
]
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_safe_get(responses):
|
||||||
|
def fake(url, accept=None):
|
||||||
|
if url not in responses:
|
||||||
|
raise BridgeError(f"Unexpected URL in test: {url}")
|
||||||
|
return json.dumps(responses[url]).encode()
|
||||||
|
|
||||||
|
return fake
|
||||||
|
|
||||||
|
|
||||||
|
class NormalizeHandleTest(SimpleTestCase):
|
||||||
|
"""Test cases for handle normalization."""
|
||||||
|
|
||||||
|
def test_valid_handle_is_lowercased_and_stripped(self):
|
||||||
|
# Given: A handle with leading @, capitals and spaces around
|
||||||
|
raw = " @Alice@Mastodon.Example "
|
||||||
|
|
||||||
|
# When: It is normalized
|
||||||
|
result = normalize_handle(raw)
|
||||||
|
|
||||||
|
# Then: The canonical lowercase form is returned
|
||||||
|
self.assertEqual(result, "alice@mastodon.example")
|
||||||
|
|
||||||
|
def test_handle_without_at_prefix_is_accepted(self):
|
||||||
|
# Given: A handle without the leading @
|
||||||
|
# When: It is normalized
|
||||||
|
# Then: It is valid
|
||||||
|
self.assertEqual(
|
||||||
|
normalize_handle("alice@mastodon.example"), "alice@mastodon.example"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_invalid_handles_return_none(self):
|
||||||
|
# Given: Handles missing parts or with invalid characters
|
||||||
|
invalid = [
|
||||||
|
"alice",
|
||||||
|
"alice@",
|
||||||
|
"@instance-only",
|
||||||
|
"alice@nodots",
|
||||||
|
"ali ce@mastodon.example",
|
||||||
|
"alice@mastodon.example/evil",
|
||||||
|
"",
|
||||||
|
None,
|
||||||
|
]
|
||||||
|
|
||||||
|
# When/Then: None of them normalize
|
||||||
|
for handle in invalid:
|
||||||
|
self.assertIsNone(normalize_handle(handle), handle)
|
||||||
|
|
||||||
|
|
||||||
|
class FetchAccountTest(SimpleTestCase):
|
||||||
|
"""Test cases for the ActivityPub account fetcher."""
|
||||||
|
|
||||||
|
def test_fetches_profile_metadata_from_actor(self):
|
||||||
|
# Given: A reachable account with actor metadata
|
||||||
|
responses = _default_responses()
|
||||||
|
|
||||||
|
# When: The account is fetched
|
||||||
|
with patch(
|
||||||
|
"app.bridge.activitypub.safe_get",
|
||||||
|
side_effect=_fake_safe_get(responses),
|
||||||
|
):
|
||||||
|
result = fetch_account("alice", "mastodon.example")
|
||||||
|
|
||||||
|
# Then: Metadata comes from the actor document
|
||||||
|
self.assertEqual(result["actor_url"], ACTOR_URL)
|
||||||
|
self.assertEqual(result["outbox_url"], OUTBOX_URL)
|
||||||
|
self.assertEqual(result["metadata"]["title"], "Alice")
|
||||||
|
self.assertEqual(result["metadata"]["nick"], "alice")
|
||||||
|
self.assertEqual(result["metadata"]["description"], "My *bio*")
|
||||||
|
self.assertEqual(
|
||||||
|
result["metadata"]["avatar"], "https://mastodon.example/avatar.png"
|
||||||
|
)
|
||||||
|
self.assertEqual(result["metadata"]["link"], "https://mastodon.example/@alice")
|
||||||
|
|
||||||
|
def test_only_top_level_notes_become_posts(self):
|
||||||
|
# Given: An outbox with a note, a reply and an announce
|
||||||
|
responses = _default_responses()
|
||||||
|
|
||||||
|
# When: The account is fetched
|
||||||
|
with patch(
|
||||||
|
"app.bridge.activitypub.safe_get",
|
||||||
|
side_effect=_fake_safe_get(responses),
|
||||||
|
):
|
||||||
|
result = fetch_account("alice", "mastodon.example")
|
||||||
|
|
||||||
|
# Then: Only the two top-level notes are bridged
|
||||||
|
self.assertEqual(len(result["posts"]), 2)
|
||||||
|
ids = [post["id"] for post in result["posts"]]
|
||||||
|
self.assertEqual(
|
||||||
|
ids, ["2025-05-02T10:00:00+00:00", "2025-05-01T10:00:00+00:00"]
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_post_carries_tags_language_cw_and_attachments(self):
|
||||||
|
# Given: Notes with hashtag, contentMap, summary and attachment
|
||||||
|
responses = _default_responses()
|
||||||
|
|
||||||
|
# When: The account is fetched
|
||||||
|
with patch(
|
||||||
|
"app.bridge.activitypub.safe_get",
|
||||||
|
side_effect=_fake_safe_get(responses),
|
||||||
|
):
|
||||||
|
result = fetch_account("alice", "mastodon.example")
|
||||||
|
|
||||||
|
# Then: The second post has tags and language
|
||||||
|
tagged = result["posts"][0]
|
||||||
|
self.assertEqual(tagged["tags"], "emacs")
|
||||||
|
self.assertEqual(tagged["language"], "es")
|
||||||
|
# And: The first post has the CW line and the attachment link
|
||||||
|
with_cw = result["posts"][1]
|
||||||
|
self.assertTrue(with_cw["content"].startswith("CW: Spoilers inside"))
|
||||||
|
self.assertIn("[[https://mastodon.example/media/cat.png]]", with_cw["content"])
|
||||||
|
|
||||||
|
def test_notes_without_valid_date_are_skipped(self):
|
||||||
|
# Given: An outbox with a dateless note and a naive-date note
|
||||||
|
responses = _default_responses()
|
||||||
|
responses[OUTBOX_PAGE_URL] = {
|
||||||
|
"orderedItems": [
|
||||||
|
{"type": "Create", "object": {"type": "Note", "content": "<p>x</p>"}},
|
||||||
|
_note_item("2025-05-01T10:00:00", "<p>naive date</p>"),
|
||||||
|
_note_item("2025-05-01T10:00:00Z", "<p>good</p>"),
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
# When: The account is fetched
|
||||||
|
with patch(
|
||||||
|
"app.bridge.activitypub.safe_get",
|
||||||
|
side_effect=_fake_safe_get(responses),
|
||||||
|
):
|
||||||
|
result = fetch_account("alice", "mastodon.example")
|
||||||
|
|
||||||
|
# Then: Only the note with a timezone-aware date survives
|
||||||
|
self.assertEqual(len(result["posts"]), 1)
|
||||||
|
self.assertEqual(result["posts"][0]["content"], "good")
|
||||||
|
|
||||||
|
def test_outbox_pagination_is_followed_up_to_max_posts(self):
|
||||||
|
# Given: An outbox split into two pages with many notes
|
||||||
|
page_two_url = "https://mastodon.example/users/alice/outbox?page=2"
|
||||||
|
responses = _default_responses()
|
||||||
|
responses[OUTBOX_PAGE_URL] = {
|
||||||
|
"orderedItems": [
|
||||||
|
_note_item(f"2025-05-02T10:00:{second:02d}Z", "<p>a</p>")
|
||||||
|
for second in range(30)
|
||||||
|
],
|
||||||
|
"next": page_two_url,
|
||||||
|
}
|
||||||
|
responses[page_two_url] = {
|
||||||
|
"orderedItems": [
|
||||||
|
_note_item(f"2025-05-01T10:00:{second:02d}Z", "<p>b</p>")
|
||||||
|
for second in range(30)
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
# When: The account is fetched
|
||||||
|
with patch(
|
||||||
|
"app.bridge.activitypub.safe_get",
|
||||||
|
side_effect=_fake_safe_get(responses),
|
||||||
|
):
|
||||||
|
result = fetch_account("alice", "mastodon.example")
|
||||||
|
|
||||||
|
# Then: Both pages are read but capped at MAX_POSTS
|
||||||
|
self.assertEqual(len(result["posts"]), MAX_POSTS)
|
||||||
|
|
||||||
|
def test_webfinger_without_actor_link_raises(self):
|
||||||
|
# Given: A WebFinger response without a self link
|
||||||
|
responses = _default_responses()
|
||||||
|
responses[WEBFINGER_URL] = {"links": [{"rel": "other"}]}
|
||||||
|
|
||||||
|
# When/Then: Fetching raises a BridgeError
|
||||||
|
with patch(
|
||||||
|
"app.bridge.activitypub.safe_get",
|
||||||
|
side_effect=_fake_safe_get(responses),
|
||||||
|
):
|
||||||
|
with self.assertRaises(BridgeError):
|
||||||
|
fetch_account("alice", "mastodon.example")
|
||||||
|
|
||||||
|
def test_invalid_json_raises_bridge_error(self):
|
||||||
|
# Given: A server answering HTML instead of JSON
|
||||||
|
def fake(url, accept=None):
|
||||||
|
return b"<html>not json</html>"
|
||||||
|
|
||||||
|
# When/Then: Fetching raises a BridgeError
|
||||||
|
with patch("app.bridge.activitypub.safe_get", side_effect=fake):
|
||||||
|
with self.assertRaises(BridgeError):
|
||||||
|
fetch_account("alice", "mastodon.example")
|
||||||
|
|
||||||
|
def test_actor_without_outbox_returns_profile_without_posts(self):
|
||||||
|
# Given: An actor document without an outbox
|
||||||
|
responses = _default_responses()
|
||||||
|
del responses[ACTOR_URL]["outbox"]
|
||||||
|
|
||||||
|
# When: The account is fetched
|
||||||
|
with patch(
|
||||||
|
"app.bridge.activitypub.safe_get",
|
||||||
|
side_effect=_fake_safe_get(responses),
|
||||||
|
):
|
||||||
|
result = fetch_account("alice", "mastodon.example")
|
||||||
|
|
||||||
|
# Then: The profile exists with no posts
|
||||||
|
self.assertEqual(result["posts"], [])
|
||||||
|
self.assertEqual(result["outbox_url"], "")
|
||||||
|
|
||||||
|
def test_outbox_with_inline_items_needs_no_extra_page(self):
|
||||||
|
# Given: An outbox embedding orderedItems directly
|
||||||
|
responses = _default_responses()
|
||||||
|
responses[OUTBOX_URL] = {
|
||||||
|
"orderedItems": [_note_item("2025-05-01T10:00:00Z", "<p>inline</p>")]
|
||||||
|
}
|
||||||
|
|
||||||
|
# When: The account is fetched
|
||||||
|
with patch(
|
||||||
|
"app.bridge.activitypub.safe_get",
|
||||||
|
side_effect=_fake_safe_get(responses),
|
||||||
|
):
|
||||||
|
result = fetch_account("alice", "mastodon.example")
|
||||||
|
|
||||||
|
# Then: The inline note is bridged
|
||||||
|
self.assertEqual(len(result["posts"]), 1)
|
||||||
|
self.assertEqual(result["posts"][0]["content"], "inline")
|
||||||
243
app/bridge/test_html_to_org.py
Normal file
243
app/bridge/test_html_to_org.py
Normal file
|
|
@ -0,0 +1,243 @@
|
||||||
|
from django.test import SimpleTestCase
|
||||||
|
|
||||||
|
from app.bridge.html_to_org import escape_org_lines, html_to_org, html_to_text
|
||||||
|
|
||||||
|
|
||||||
|
class HtmlToOrgTest(SimpleTestCase):
|
||||||
|
"""Test cases for the HTML to Org converter."""
|
||||||
|
|
||||||
|
def test_paragraphs_become_blank_line_separated(self):
|
||||||
|
# Given: HTML with two paragraphs
|
||||||
|
html = "<p>First paragraph</p><p>Second paragraph</p>"
|
||||||
|
|
||||||
|
# When: It is converted to Org
|
||||||
|
result = html_to_org(html)
|
||||||
|
|
||||||
|
# Then: Paragraphs are separated by a blank line
|
||||||
|
self.assertEqual(result, "First paragraph\n\nSecond paragraph")
|
||||||
|
|
||||||
|
def test_br_becomes_newline(self):
|
||||||
|
# Given: HTML with a line break inside a paragraph
|
||||||
|
html = "<p>First line<br>Second line</p>"
|
||||||
|
|
||||||
|
# When: It is converted to Org
|
||||||
|
result = html_to_org(html)
|
||||||
|
|
||||||
|
# Then: The break is kept as a single newline
|
||||||
|
self.assertEqual(result, "First line\nSecond line")
|
||||||
|
|
||||||
|
def test_link_with_text_becomes_org_link(self):
|
||||||
|
# Given: An anchor with its own text
|
||||||
|
html = '<p>Read <a href="https://example.com/post">this article</a></p>'
|
||||||
|
|
||||||
|
# When: It is converted to Org
|
||||||
|
result = html_to_org(html)
|
||||||
|
|
||||||
|
# Then: An Org link with description is produced
|
||||||
|
self.assertEqual(result, "Read [[https://example.com/post][this article]]")
|
||||||
|
|
||||||
|
def test_link_with_url_as_text_becomes_plain_org_link(self):
|
||||||
|
# Given: An anchor whose text is the URL itself
|
||||||
|
html = '<a href="https://example.com">https://example.com</a>'
|
||||||
|
|
||||||
|
# When: It is converted to Org
|
||||||
|
result = html_to_org(html)
|
||||||
|
|
||||||
|
# Then: A plain Org link is produced
|
||||||
|
self.assertEqual(result, "[[https://example.com]]")
|
||||||
|
|
||||||
|
def test_hashtag_and_mention_links_stay_as_plain_text(self):
|
||||||
|
# Given: Mastodon-style hashtag and mention anchors
|
||||||
|
html = (
|
||||||
|
'<p>Hi <a href="https://m.example/@bob" class="u-url mention">'
|
||||||
|
"@bob</a> about "
|
||||||
|
'<a href="https://m.example/tags/emacs" class="mention hashtag">'
|
||||||
|
"#emacs</a></p>"
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: It is converted to Org
|
||||||
|
result = html_to_org(html)
|
||||||
|
|
||||||
|
# Then: No Org links are produced, only their text
|
||||||
|
self.assertEqual(result, "Hi @bob about #emacs")
|
||||||
|
|
||||||
|
def test_emphasis_tags_become_org_markers(self):
|
||||||
|
# Given: HTML with bold, italic and code
|
||||||
|
html = "<p><strong>bold</strong> <em>italic</em> <code>code</code></p>"
|
||||||
|
|
||||||
|
# When: It is converted to Org
|
||||||
|
result = html_to_org(html)
|
||||||
|
|
||||||
|
# Then: Org emphasis markers are used
|
||||||
|
self.assertEqual(result, "*bold* /italic/ ~code~")
|
||||||
|
|
||||||
|
def test_list_items_become_org_list(self):
|
||||||
|
# Given: An unordered list
|
||||||
|
html = "<ul><li>One</li><li>Two</li></ul>"
|
||||||
|
|
||||||
|
# When: It is converted to Org
|
||||||
|
result = html_to_org(html)
|
||||||
|
|
||||||
|
# Then: Each item becomes a dash line
|
||||||
|
self.assertEqual(result, "- One\n- Two")
|
||||||
|
|
||||||
|
def test_blockquote_becomes_quote_block(self):
|
||||||
|
# Given: A quoted paragraph
|
||||||
|
html = "<blockquote><p>Wise words</p></blockquote>"
|
||||||
|
|
||||||
|
# When: It is converted to Org
|
||||||
|
result = html_to_org(html)
|
||||||
|
|
||||||
|
# Then: The text is wrapped in a quote block
|
||||||
|
self.assertEqual(result, "#+BEGIN_QUOTE\n\nWise words\n\n#+END_QUOTE")
|
||||||
|
|
||||||
|
def test_pre_becomes_example_block(self):
|
||||||
|
# Given: Preformatted content
|
||||||
|
html = "<pre>print('hi')</pre>"
|
||||||
|
|
||||||
|
# When: It is converted to Org
|
||||||
|
result = html_to_org(html)
|
||||||
|
|
||||||
|
# Then: The content is wrapped in an example block preserving text
|
||||||
|
self.assertEqual(result, "#+BEGIN_EXAMPLE\nprint('hi')\n#+END_EXAMPLE")
|
||||||
|
|
||||||
|
def test_invisible_spans_are_skipped(self):
|
||||||
|
# Given: A Mastodon-style shortened URL anchor
|
||||||
|
html = (
|
||||||
|
'<a href="https://example.com/very/long/path">'
|
||||||
|
'<span class="invisible">https://</span>'
|
||||||
|
'<span class="ellipsis">example.com/very</span></a>'
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: It is converted to Org
|
||||||
|
result = html_to_org(html)
|
||||||
|
|
||||||
|
# Then: The invisible prefix is not part of the link text
|
||||||
|
self.assertEqual(
|
||||||
|
result, "[[https://example.com/very/long/path][example.com/very]]"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_image_becomes_org_link_with_alt_text(self):
|
||||||
|
# Given: An image with alt text and another without it
|
||||||
|
html = (
|
||||||
|
'<p><img src="https://example.com/a.png" alt="A comic"/></p>'
|
||||||
|
'<p><img src="https://example.com/b.png"/></p>'
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: It is converted to Org
|
||||||
|
result = html_to_org(html)
|
||||||
|
|
||||||
|
# Then: Images become Org links, using alt as description
|
||||||
|
self.assertEqual(
|
||||||
|
result,
|
||||||
|
"[[https://example.com/a.png][A comic]]\n\n[[https://example.com/b.png]]",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_image_wrapped_in_anchor_keeps_only_the_image_link(self):
|
||||||
|
# Given: An image wrapped in an anchor (common in RSS feeds)
|
||||||
|
html = (
|
||||||
|
'<a href="https://example.com/post">'
|
||||||
|
'<img src="https://example.com/a.png" alt="A comic"/></a>'
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: It is converted to Org
|
||||||
|
result = html_to_org(html)
|
||||||
|
|
||||||
|
# Then: No nested Org links are produced
|
||||||
|
self.assertEqual(result, "[[https://example.com/a.png][A comic]]")
|
||||||
|
|
||||||
|
def test_html_entities_are_decoded(self):
|
||||||
|
# Given: HTML with entities
|
||||||
|
html = "<p>Fish & chips <3</p>"
|
||||||
|
|
||||||
|
# When: It is converted to Org
|
||||||
|
result = html_to_org(html)
|
||||||
|
|
||||||
|
# Then: Entities are decoded
|
||||||
|
self.assertEqual(result, "Fish & chips <3")
|
||||||
|
|
||||||
|
def test_headline_injection_is_escaped(self):
|
||||||
|
# Given: Remote content that looks like Org Social headlines
|
||||||
|
html = "<p>** 2025-01-01T00:00:00+00:00<br>* Posts</p>"
|
||||||
|
|
||||||
|
# When: It is converted to Org
|
||||||
|
result = html_to_org(html)
|
||||||
|
|
||||||
|
# Then: The lines are prefixed so they cannot break the feed
|
||||||
|
for line in result.split("\n"):
|
||||||
|
self.assertFalse(line.startswith("*"))
|
||||||
|
|
||||||
|
def test_empty_and_missing_html_return_empty_string(self):
|
||||||
|
# Given: Empty and missing input
|
||||||
|
# When: They are converted
|
||||||
|
# Then: The result is an empty string
|
||||||
|
self.assertEqual(html_to_org(""), "")
|
||||||
|
self.assertEqual(html_to_org(None), "")
|
||||||
|
|
||||||
|
def test_unknown_tags_are_stripped_keeping_text(self):
|
||||||
|
# Given: HTML with tags the converter does not know
|
||||||
|
html = "<article><section>Some text</section></article>"
|
||||||
|
|
||||||
|
# When: It is converted to Org
|
||||||
|
result = html_to_org(html)
|
||||||
|
|
||||||
|
# Then: The text survives without the tags
|
||||||
|
self.assertEqual(result, "Some text")
|
||||||
|
|
||||||
|
|
||||||
|
class HtmlToTextTest(SimpleTestCase):
|
||||||
|
"""Test cases for the single-line text converter."""
|
||||||
|
|
||||||
|
def test_multiline_html_becomes_single_line(self):
|
||||||
|
# Given: HTML with several paragraphs
|
||||||
|
html = "<p>Bio line one</p><p>Bio line two</p>"
|
||||||
|
|
||||||
|
# When: It is converted to plain text
|
||||||
|
result = html_to_text(html)
|
||||||
|
|
||||||
|
# Then: Everything is collapsed into one line
|
||||||
|
self.assertEqual(result, "Bio line one Bio line two")
|
||||||
|
|
||||||
|
|
||||||
|
class EscapeOrgLinesTest(SimpleTestCase):
|
||||||
|
"""Test cases for headline escaping."""
|
||||||
|
|
||||||
|
def test_one_and_two_asterisk_headlines_are_escaped(self):
|
||||||
|
# Given: Text with dangerous headline-like lines
|
||||||
|
text = "* Posts\n** 2025-01-01T00:00:00+00:00\nnormal"
|
||||||
|
|
||||||
|
# When: It is escaped
|
||||||
|
result = escape_org_lines(text)
|
||||||
|
|
||||||
|
# Then: Dangerous lines are prefixed with a space
|
||||||
|
self.assertEqual(result, " * Posts\n ** 2025-01-01T00:00:00+00:00\nnormal")
|
||||||
|
|
||||||
|
def test_post_titles_with_three_asterisks_are_kept(self):
|
||||||
|
# Given: A legal Org Social post title line
|
||||||
|
text = "*** My title"
|
||||||
|
|
||||||
|
# When: It is escaped
|
||||||
|
result = escape_org_lines(text)
|
||||||
|
|
||||||
|
# Then: The title line is untouched
|
||||||
|
self.assertEqual(result, "*** My title")
|
||||||
|
|
||||||
|
def test_bold_text_at_line_start_is_kept(self):
|
||||||
|
# Given: A line starting with Org bold (no space after asterisk)
|
||||||
|
text = "*bold* rest"
|
||||||
|
|
||||||
|
# When: It is escaped
|
||||||
|
result = escape_org_lines(text)
|
||||||
|
|
||||||
|
# Then: The line is untouched
|
||||||
|
self.assertEqual(result, "*bold* rest")
|
||||||
|
|
||||||
|
def test_escaping_is_idempotent(self):
|
||||||
|
# Given: Already escaped text
|
||||||
|
text = escape_org_lines("** 2025-01-01T00:00:00+00:00")
|
||||||
|
|
||||||
|
# When: It is escaped again
|
||||||
|
result = escape_org_lines(text)
|
||||||
|
|
||||||
|
# Then: Nothing changes
|
||||||
|
self.assertEqual(result, text)
|
||||||
42
app/bridge/test_models.py
Normal file
42
app/bridge/test_models.py
Normal file
|
|
@ -0,0 +1,42 @@
|
||||||
|
from django.test import SimpleTestCase
|
||||||
|
|
||||||
|
from app.bridge.models import is_bridge_feed_url
|
||||||
|
|
||||||
|
|
||||||
|
class IsBridgeFeedUrlTest(SimpleTestCase):
|
||||||
|
"""Test cases for is_bridge_feed_url using Given/When/Then structure."""
|
||||||
|
|
||||||
|
def test_detects_rss_bridge_url(self):
|
||||||
|
"""Test an RSS bridge URL is detected as a bridge feed."""
|
||||||
|
# Given: The URL of an RSS bridge on any relay
|
||||||
|
url = (
|
||||||
|
"https://relay.org-social.org/bridge/rss/"
|
||||||
|
"?url=https%3A%2F%2Frss.arxiv.org%2Frss%2Fquant-ph"
|
||||||
|
)
|
||||||
|
|
||||||
|
# When / Then: It is identified as a bridge feed
|
||||||
|
self.assertTrue(is_bridge_feed_url(url))
|
||||||
|
|
||||||
|
def test_detects_activitypub_bridge_url(self):
|
||||||
|
"""Test an ActivityPub bridge URL is detected as a bridge feed."""
|
||||||
|
# Given: The URL of an ActivityPub bridge on any relay
|
||||||
|
url = "https://other-relay.org/bridge/activitypub/@user@instance.tld/"
|
||||||
|
|
||||||
|
# When / Then: It is identified as a bridge feed
|
||||||
|
self.assertTrue(is_bridge_feed_url(url))
|
||||||
|
|
||||||
|
def test_regular_feed_is_not_a_bridge(self):
|
||||||
|
"""Test a regular social.org feed is not detected as a bridge."""
|
||||||
|
# Given: The URL of a real Org Social feed
|
||||||
|
url = "https://example.com/social.org"
|
||||||
|
|
||||||
|
# When / Then: It is not identified as a bridge feed
|
||||||
|
self.assertFalse(is_bridge_feed_url(url))
|
||||||
|
|
||||||
|
def test_bridge_marker_in_query_string_is_ignored(self):
|
||||||
|
"""Test the bridge marker only matches in the URL path."""
|
||||||
|
# Given: A real feed whose query string mentions a bridge path
|
||||||
|
url = "https://example.com/social.org?note=/bridge/rss/"
|
||||||
|
|
||||||
|
# When / Then: It is not identified as a bridge feed
|
||||||
|
self.assertFalse(is_bridge_feed_url(url))
|
||||||
118
app/bridge/test_org_renderer.py
Normal file
118
app/bridge/test_org_renderer.py
Normal file
|
|
@ -0,0 +1,118 @@
|
||||||
|
from django.test import TestCase
|
||||||
|
|
||||||
|
from app.bridge.org_renderer import render_profile_org
|
||||||
|
from app.feeds.models import Post, Profile, ProfileLink
|
||||||
|
|
||||||
|
|
||||||
|
class RenderProfileOrgTest(TestCase):
|
||||||
|
"""Test cases for the virtual social.org renderer."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
# Given: A bridged profile with a link and two posts
|
||||||
|
self.profile = Profile.objects.create(
|
||||||
|
feed="http://localhost:8080/bridge/activitypub/@alice@m.example/",
|
||||||
|
title="Alice in Wonderland",
|
||||||
|
nick="alice",
|
||||||
|
description="Just a test account",
|
||||||
|
avatar="https://m.example/avatar.png",
|
||||||
|
version="v1",
|
||||||
|
)
|
||||||
|
ProfileLink.objects.create(profile=self.profile, url="https://m.example/@alice")
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile,
|
||||||
|
post_id="2025-05-01T10:00:00+00:00",
|
||||||
|
content="Hello world",
|
||||||
|
created_at="2025-05-01T10:00:00+00:00",
|
||||||
|
)
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile,
|
||||||
|
post_id="2025-05-02T10:00:00+00:00",
|
||||||
|
content="Second post",
|
||||||
|
tags="emacs org",
|
||||||
|
language="en",
|
||||||
|
created_at="2025-05-02T10:00:00+00:00",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_renders_global_metadata(self):
|
||||||
|
# When: The profile is rendered
|
||||||
|
result = render_profile_org(self.profile, self.profile.posts.all())
|
||||||
|
|
||||||
|
# Then: All global metadata lines are present
|
||||||
|
self.assertIn("#+TITLE: Alice in Wonderland\n", result)
|
||||||
|
self.assertIn("#+NICK: alice\n", result)
|
||||||
|
self.assertIn("#+DESCRIPTION: Just a test account\n", result)
|
||||||
|
self.assertIn("#+AVATAR: https://m.example/avatar.png\n", result)
|
||||||
|
self.assertIn("#+LINK: https://m.example/@alice\n", result)
|
||||||
|
|
||||||
|
def test_renders_posts_section_with_properties(self):
|
||||||
|
# When: The profile is rendered oldest post first
|
||||||
|
posts = self.profile.posts.order_by("created_at")
|
||||||
|
result = render_profile_org(self.profile, posts)
|
||||||
|
|
||||||
|
# Then: Posts appear under * Posts with their properties
|
||||||
|
self.assertIn("* Posts\n", result)
|
||||||
|
self.assertIn("** 2025-05-01T10:00:00+00:00\n", result)
|
||||||
|
self.assertIn("** 2025-05-02T10:00:00+00:00\n", result)
|
||||||
|
self.assertIn(":TAGS: emacs org\n", result)
|
||||||
|
self.assertIn(":LANG: en\n", result)
|
||||||
|
self.assertIn("Hello world\n", result)
|
||||||
|
# And: The first post comes before the second
|
||||||
|
self.assertLess(
|
||||||
|
result.index("2025-05-01T10:00:00+00:00"),
|
||||||
|
result.index("2025-05-02T10:00:00+00:00"),
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_nick_with_spaces_is_sanitized(self):
|
||||||
|
# Given: A profile whose nick contains spaces
|
||||||
|
self.profile.nick = "alice cooper"
|
||||||
|
|
||||||
|
# When: The profile is rendered
|
||||||
|
result = render_profile_org(self.profile, [])
|
||||||
|
|
||||||
|
# Then: The nick has no spaces
|
||||||
|
self.assertIn("#+NICK: alice_cooper\n", result)
|
||||||
|
|
||||||
|
def test_metadata_never_spans_multiple_lines(self):
|
||||||
|
# Given: A profile description containing newlines
|
||||||
|
self.profile.description = "line one\nline two"
|
||||||
|
|
||||||
|
# When: The profile is rendered
|
||||||
|
result = render_profile_org(self.profile, [])
|
||||||
|
|
||||||
|
# Then: The description is collapsed into a single line
|
||||||
|
self.assertIn("#+DESCRIPTION: line one line two\n", result)
|
||||||
|
|
||||||
|
def test_post_content_headlines_are_escaped(self):
|
||||||
|
# Given: A post whose content looks like an Org Social headline
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile,
|
||||||
|
post_id="2025-05-03T10:00:00+00:00",
|
||||||
|
content="** 2030-01-01T00:00:00+00:00\ninjected",
|
||||||
|
created_at="2025-05-03T10:00:00+00:00",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The profile is rendered
|
||||||
|
result = render_profile_org(
|
||||||
|
self.profile, self.profile.posts.order_by("created_at")
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: The injected headline is escaped
|
||||||
|
self.assertIn(" ** 2030-01-01T00:00:00+00:00\n", result)
|
||||||
|
self.assertNotIn("\n** 2030-01-01T00:00:00+00:00\n", result)
|
||||||
|
|
||||||
|
def test_empty_optional_metadata_is_omitted(self):
|
||||||
|
# Given: A profile with no description, avatar nor links
|
||||||
|
profile = Profile.objects.create(
|
||||||
|
feed="http://localhost:8080/bridge/rss/?url=x",
|
||||||
|
title="Bare",
|
||||||
|
nick="bare",
|
||||||
|
version="v1",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The profile is rendered
|
||||||
|
result = render_profile_org(profile, [])
|
||||||
|
|
||||||
|
# Then: Optional lines are not present
|
||||||
|
self.assertNotIn("#+DESCRIPTION:", result)
|
||||||
|
self.assertNotIn("#+AVATAR:", result)
|
||||||
|
self.assertNotIn("#+LINK:", result)
|
||||||
172
app/bridge/test_rss.py
Normal file
172
app/bridge/test_rss.py
Normal file
|
|
@ -0,0 +1,172 @@
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from django.test import SimpleTestCase
|
||||||
|
|
||||||
|
from app.bridge.fetching import BridgeError
|
||||||
|
from app.bridge.rss import fetch_rss_feed
|
||||||
|
|
||||||
|
RSS_SAMPLE = b"""<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<rss version="2.0">
|
||||||
|
<channel>
|
||||||
|
<title>My Blog</title>
|
||||||
|
<link>https://blog.example</link>
|
||||||
|
<description>Notes about software</description>
|
||||||
|
<language>en-us</language>
|
||||||
|
<image>
|
||||||
|
<url>https://blog.example/logo.png</url>
|
||||||
|
<title>My Blog</title>
|
||||||
|
<link>https://blog.example</link>
|
||||||
|
</image>
|
||||||
|
<item>
|
||||||
|
<title>Second article</title>
|
||||||
|
<link>https://blog.example/second</link>
|
||||||
|
<pubDate>Fri, 02 May 2025 10:00:00 GMT</pubDate>
|
||||||
|
<category>emacs</category>
|
||||||
|
<category>org mode</category>
|
||||||
|
<description><p>Second <strong>body</strong></p></description>
|
||||||
|
</item>
|
||||||
|
<item>
|
||||||
|
<title>First article</title>
|
||||||
|
<link>https://blog.example/first</link>
|
||||||
|
<pubDate>Thu, 01 May 2025 10:00:00 GMT</pubDate>
|
||||||
|
<description>First body</description>
|
||||||
|
</item>
|
||||||
|
<item>
|
||||||
|
<title>No date article</title>
|
||||||
|
<link>https://blog.example/undated</link>
|
||||||
|
<description>Should be skipped</description>
|
||||||
|
</item>
|
||||||
|
</channel>
|
||||||
|
</rss>
|
||||||
|
"""
|
||||||
|
|
||||||
|
ATOM_SAMPLE = b"""<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<feed xmlns="http://www.w3.org/2005/Atom">
|
||||||
|
<title>Atom Feed</title>
|
||||||
|
<subtitle>An atom feed</subtitle>
|
||||||
|
<link href="https://atom.example"/>
|
||||||
|
<entry>
|
||||||
|
<title>Atom entry</title>
|
||||||
|
<link href="https://atom.example/entry"/>
|
||||||
|
<updated>2025-05-01T10:00:00Z</updated>
|
||||||
|
<content type="html"><p>Atom body</p></content>
|
||||||
|
</entry>
|
||||||
|
</feed>
|
||||||
|
"""
|
||||||
|
|
||||||
|
DUPLICATED_DATES_SAMPLE = b"""<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<rss version="2.0">
|
||||||
|
<channel>
|
||||||
|
<title>Daily</title>
|
||||||
|
<link>https://daily.example</link>
|
||||||
|
<item>
|
||||||
|
<title>Post A</title>
|
||||||
|
<link>https://daily.example/a</link>
|
||||||
|
<pubDate>Thu, 01 May 2025 00:00:00 GMT</pubDate>
|
||||||
|
</item>
|
||||||
|
<item>
|
||||||
|
<title>Post B</title>
|
||||||
|
<link>https://daily.example/b</link>
|
||||||
|
<pubDate>Thu, 01 May 2025 00:00:00 GMT</pubDate>
|
||||||
|
</item>
|
||||||
|
</channel>
|
||||||
|
</rss>
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
class FetchRssFeedTest(SimpleTestCase):
|
||||||
|
"""Test cases for the RSS/Atom fetcher."""
|
||||||
|
|
||||||
|
def test_rss2_metadata_is_extracted(self):
|
||||||
|
# Given: A standard RSS 2.0 feed
|
||||||
|
with patch("app.bridge.rss.safe_get", return_value=RSS_SAMPLE):
|
||||||
|
# When: The feed is fetched
|
||||||
|
result = fetch_rss_feed("https://blog.example/feed.xml")
|
||||||
|
|
||||||
|
# Then: The channel metadata maps to profile metadata
|
||||||
|
metadata = result["metadata"]
|
||||||
|
self.assertEqual(metadata["title"], "My Blog")
|
||||||
|
self.assertEqual(metadata["nick"], "My_Blog")
|
||||||
|
self.assertEqual(metadata["description"], "Notes about software")
|
||||||
|
self.assertEqual(metadata["avatar"], "https://blog.example/logo.png")
|
||||||
|
self.assertEqual(metadata["link"], "https://blog.example")
|
||||||
|
self.assertEqual(metadata["language"], "en")
|
||||||
|
|
||||||
|
def test_entries_become_posts_with_title_body_and_link(self):
|
||||||
|
# Given: A standard RSS 2.0 feed
|
||||||
|
with patch("app.bridge.rss.safe_get", return_value=RSS_SAMPLE):
|
||||||
|
# When: The feed is fetched
|
||||||
|
result = fetch_rss_feed("https://blog.example/feed.xml")
|
||||||
|
|
||||||
|
# Then: Dated entries become posts with UTC IDs
|
||||||
|
ids = [post["id"] for post in result["posts"]]
|
||||||
|
self.assertEqual(
|
||||||
|
ids, ["2025-05-02T10:00:00+00:00", "2025-05-01T10:00:00+00:00"]
|
||||||
|
)
|
||||||
|
# And: The body has the title heading, converted content and link
|
||||||
|
second = result["posts"][0]
|
||||||
|
self.assertIn("*** Second article", second["content"])
|
||||||
|
self.assertIn("Second *body*", second["content"])
|
||||||
|
self.assertIn("[[https://blog.example/second]]", second["content"])
|
||||||
|
# And: Categories become tags with spaces replaced
|
||||||
|
self.assertEqual(second["tags"], "emacs org-mode")
|
||||||
|
|
||||||
|
def test_entries_without_date_are_skipped(self):
|
||||||
|
# Given: A feed with a dateless entry
|
||||||
|
with patch("app.bridge.rss.safe_get", return_value=RSS_SAMPLE):
|
||||||
|
# When: The feed is fetched
|
||||||
|
result = fetch_rss_feed("https://blog.example/feed.xml")
|
||||||
|
|
||||||
|
# Then: The dateless entry is not bridged
|
||||||
|
contents = " ".join(post["content"] for post in result["posts"])
|
||||||
|
self.assertNotIn("No date article", contents)
|
||||||
|
|
||||||
|
def test_atom_feeds_are_supported(self):
|
||||||
|
# Given: An Atom feed
|
||||||
|
with patch("app.bridge.rss.safe_get", return_value=ATOM_SAMPLE):
|
||||||
|
# When: The feed is fetched
|
||||||
|
result = fetch_rss_feed("https://atom.example/feed.atom")
|
||||||
|
|
||||||
|
# Then: Metadata and the entry are bridged
|
||||||
|
self.assertEqual(result["metadata"]["title"], "Atom Feed")
|
||||||
|
self.assertEqual(len(result["posts"]), 1)
|
||||||
|
self.assertEqual(result["posts"][0]["id"], "2025-05-01T10:00:00+00:00")
|
||||||
|
self.assertIn("Atom body", result["posts"][0]["content"])
|
||||||
|
|
||||||
|
def test_entries_with_same_date_get_shifted_ids(self):
|
||||||
|
# Given: A feed where two entries share the exact same date
|
||||||
|
with patch("app.bridge.rss.safe_get", return_value=DUPLICATED_DATES_SAMPLE):
|
||||||
|
# When: The feed is fetched
|
||||||
|
result = fetch_rss_feed("https://daily.example/feed.xml")
|
||||||
|
|
||||||
|
# Then: Both entries survive with distinct consecutive IDs
|
||||||
|
ids = sorted(post["id"] for post in result["posts"])
|
||||||
|
self.assertEqual(
|
||||||
|
ids, ["2025-05-01T00:00:00+00:00", "2025-05-01T00:00:01+00:00"]
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_non_feed_content_raises_bridge_error(self):
|
||||||
|
# Given: A URL answering plain HTML
|
||||||
|
with patch(
|
||||||
|
"app.bridge.rss.safe_get", return_value=b"<html><body>hi</body></html>"
|
||||||
|
):
|
||||||
|
# When/Then: Fetching raises a BridgeError
|
||||||
|
with self.assertRaises(BridgeError):
|
||||||
|
fetch_rss_feed("https://blog.example/not-a-feed")
|
||||||
|
|
||||||
|
def test_feed_without_title_uses_host_as_nick(self):
|
||||||
|
# Given: A minimal feed without a channel title
|
||||||
|
sample = (
|
||||||
|
b'<?xml version="1.0"?><rss version="2.0"><channel>'
|
||||||
|
b"<item><title>x</title>"
|
||||||
|
b"<pubDate>Thu, 01 May 2025 00:00:00 GMT</pubDate></item>"
|
||||||
|
b"</channel></rss>"
|
||||||
|
)
|
||||||
|
with patch("app.bridge.rss.safe_get", return_value=sample):
|
||||||
|
# When: The feed is fetched
|
||||||
|
result = fetch_rss_feed("https://blog.example/feed.xml")
|
||||||
|
|
||||||
|
# Then: The nick is derived from the host
|
||||||
|
self.assertEqual(result["metadata"]["nick"], "blog_example")
|
||||||
|
# And: The title falls back to the source URL
|
||||||
|
self.assertEqual(result["metadata"]["title"], "https://blog.example/feed.xml")
|
||||||
166
app/bridge/test_store.py
Normal file
166
app/bridge/test_store.py
Normal file
|
|
@ -0,0 +1,166 @@
|
||||||
|
from django.test import TestCase
|
||||||
|
|
||||||
|
from app.bridge.store import store_bridge_data
|
||||||
|
from app.feeds.models import Post, Profile
|
||||||
|
|
||||||
|
FEED_URL = "http://localhost:8080/bridge/activitypub/@alice@m.example/"
|
||||||
|
|
||||||
|
|
||||||
|
def _data(posts=None, nick="alice", title="Alice"):
|
||||||
|
return {
|
||||||
|
"metadata": {
|
||||||
|
"title": title,
|
||||||
|
"nick": nick,
|
||||||
|
"description": "Bio",
|
||||||
|
"avatar": "https://m.example/avatar.png",
|
||||||
|
"link": "https://m.example/@alice",
|
||||||
|
},
|
||||||
|
"posts": posts if posts is not None else [],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _post(post_id, content, tags=""):
|
||||||
|
return {"id": post_id, "content": content, "tags": tags, "language": ""}
|
||||||
|
|
||||||
|
|
||||||
|
class StoreBridgeDataTest(TestCase):
|
||||||
|
"""Test cases for storing normalized bridge data."""
|
||||||
|
|
||||||
|
def test_creates_profile_with_posts_and_link(self):
|
||||||
|
# Given: Normalized data with two posts
|
||||||
|
data = _data(
|
||||||
|
posts=[
|
||||||
|
_post("2025-05-01T10:00:00+00:00", "First"),
|
||||||
|
_post("2025-05-02T10:00:00+00:00", "Second", tags="emacs"),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The data is stored
|
||||||
|
profile = store_bridge_data(FEED_URL, data)
|
||||||
|
|
||||||
|
# Then: Profile, posts and link exist in the database
|
||||||
|
self.assertEqual(profile.feed, FEED_URL)
|
||||||
|
self.assertEqual(profile.nick, "alice")
|
||||||
|
self.assertEqual(profile.posts.count(), 2)
|
||||||
|
self.assertEqual(
|
||||||
|
list(profile.links.values_list("url", flat=True)),
|
||||||
|
["https://m.example/@alice"],
|
||||||
|
)
|
||||||
|
post = profile.posts.get(post_id="2025-05-02T10:00:00+00:00")
|
||||||
|
self.assertEqual(post.tags, "emacs")
|
||||||
|
|
||||||
|
def test_nick_with_spaces_is_sanitized(self):
|
||||||
|
# Given: Metadata with a nick containing spaces
|
||||||
|
data = _data(nick="alice cooper")
|
||||||
|
|
||||||
|
# When: The data is stored
|
||||||
|
profile = store_bridge_data(FEED_URL, data)
|
||||||
|
|
||||||
|
# Then: The nick has no spaces
|
||||||
|
self.assertEqual(profile.nick, "alice_cooper")
|
||||||
|
|
||||||
|
def test_unchanged_data_is_not_rewritten(self):
|
||||||
|
# Given: Data already stored once
|
||||||
|
data = _data(posts=[_post("2025-05-01T10:00:00+00:00", "First")])
|
||||||
|
profile = store_bridge_data(FEED_URL, data)
|
||||||
|
version = profile.version
|
||||||
|
|
||||||
|
# When: The exact same data is stored again
|
||||||
|
profile_again = store_bridge_data(FEED_URL, data)
|
||||||
|
|
||||||
|
# Then: The version hash does not change and posts are intact
|
||||||
|
self.assertEqual(profile_again.version, version)
|
||||||
|
self.assertEqual(profile_again.posts.count(), 1)
|
||||||
|
|
||||||
|
def test_changed_post_content_is_updated(self):
|
||||||
|
# Given: A stored post whose content changes at the origin
|
||||||
|
post_id = "2025-05-01T10:00:00+00:00"
|
||||||
|
store_bridge_data(FEED_URL, _data(posts=[_post(post_id, "Original")]))
|
||||||
|
|
||||||
|
# When: The new version is stored
|
||||||
|
store_bridge_data(FEED_URL, _data(posts=[_post(post_id, "Edited")]))
|
||||||
|
|
||||||
|
# Then: The post content is updated in place
|
||||||
|
profile = Profile.objects.get(feed=FEED_URL)
|
||||||
|
self.assertEqual(profile.posts.get(post_id=post_id).content, "Edited")
|
||||||
|
self.assertEqual(profile.posts.count(), 1)
|
||||||
|
|
||||||
|
def test_posts_deleted_at_origin_are_removed_within_window(self):
|
||||||
|
# Given: Three stored posts
|
||||||
|
store_bridge_data(
|
||||||
|
FEED_URL,
|
||||||
|
_data(
|
||||||
|
posts=[
|
||||||
|
_post("2025-05-01T10:00:00+00:00", "A"),
|
||||||
|
_post("2025-05-02T10:00:00+00:00", "B"),
|
||||||
|
_post("2025-05-03T10:00:00+00:00", "C"),
|
||||||
|
]
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The origin no longer returns the middle post
|
||||||
|
store_bridge_data(
|
||||||
|
FEED_URL,
|
||||||
|
_data(
|
||||||
|
posts=[
|
||||||
|
_post("2025-05-01T10:00:00+00:00", "A"),
|
||||||
|
_post("2025-05-03T10:00:00+00:00", "C"),
|
||||||
|
]
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: The deleted post is removed
|
||||||
|
profile = Profile.objects.get(feed=FEED_URL)
|
||||||
|
ids = set(profile.posts.values_list("post_id", flat=True))
|
||||||
|
self.assertEqual(
|
||||||
|
ids, {"2025-05-01T10:00:00+00:00", "2025-05-03T10:00:00+00:00"}
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_posts_older_than_fetched_window_are_preserved(self):
|
||||||
|
# Given: An old post stored from a previous, deeper fetch
|
||||||
|
store_bridge_data(
|
||||||
|
FEED_URL, _data(posts=[_post("2025-01-01T10:00:00+00:00", "Old")])
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: A later fetch only returns newer posts
|
||||||
|
store_bridge_data(
|
||||||
|
FEED_URL, _data(posts=[_post("2025-05-01T10:00:00+00:00", "New")])
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: The old post outside the fetched window survives
|
||||||
|
profile = Profile.objects.get(feed=FEED_URL)
|
||||||
|
ids = set(profile.posts.values_list("post_id", flat=True))
|
||||||
|
self.assertEqual(
|
||||||
|
ids, {"2025-01-01T10:00:00+00:00", "2025-05-01T10:00:00+00:00"}
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_no_webmentions_are_queued_for_bridged_posts(self):
|
||||||
|
# Given: A bridged post containing an external link
|
||||||
|
from app.feeds.models import OutgoingWebmention
|
||||||
|
|
||||||
|
data = _data(
|
||||||
|
posts=[
|
||||||
|
_post(
|
||||||
|
"2025-05-01T10:00:00+00:00",
|
||||||
|
"Look at [[https://external.example/page][this]]",
|
||||||
|
)
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The data is stored
|
||||||
|
store_bridge_data(FEED_URL, data)
|
||||||
|
|
||||||
|
# Then: No outgoing webmention is created
|
||||||
|
self.assertEqual(OutgoingWebmention.objects.count(), 0)
|
||||||
|
|
||||||
|
def test_post_created_at_comes_from_post_id(self):
|
||||||
|
# Given: A post with a known timestamp ID
|
||||||
|
store_bridge_data(
|
||||||
|
FEED_URL, _data(posts=[_post("2025-05-01T10:00:00+00:00", "A")])
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The post is read back
|
||||||
|
post = Post.objects.get(post_id="2025-05-01T10:00:00+00:00")
|
||||||
|
|
||||||
|
# Then: created_at matches the ID timestamp
|
||||||
|
self.assertEqual(post.created_at.isoformat(), "2025-05-01T10:00:00+00:00")
|
||||||
160
app/bridge/test_tasks.py
Normal file
160
app/bridge/test_tasks.py
Normal file
|
|
@ -0,0 +1,160 @@
|
||||||
|
from datetime import timedelta
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from django.test import TestCase
|
||||||
|
from django.utils import timezone
|
||||||
|
|
||||||
|
from app.bridge.fetching import BridgeError
|
||||||
|
from app.bridge.models import (
|
||||||
|
BridgedActivityPubAccount,
|
||||||
|
BridgedRssFeed,
|
||||||
|
activitypub_feed_url,
|
||||||
|
rss_feed_url,
|
||||||
|
)
|
||||||
|
from app.bridge.tasks import (
|
||||||
|
BRIDGE_ACTIVE_DAYS,
|
||||||
|
BRIDGE_STALE_DAYS,
|
||||||
|
_cleanup_stale_bridges_impl,
|
||||||
|
_refresh_bridges_impl,
|
||||||
|
)
|
||||||
|
from app.feeds.models import Post, Profile
|
||||||
|
|
||||||
|
|
||||||
|
def _make_ap_bridge(handle, accessed_days_ago=0):
|
||||||
|
profile = Profile.objects.create(
|
||||||
|
feed=activitypub_feed_url(handle),
|
||||||
|
title=handle,
|
||||||
|
nick=handle.split("@")[0],
|
||||||
|
version="v1",
|
||||||
|
)
|
||||||
|
return BridgedActivityPubAccount.objects.create(
|
||||||
|
handle=handle,
|
||||||
|
actor_url=f"https://{handle.split('@')[1]}/users/{handle.split('@')[0]}",
|
||||||
|
profile=profile,
|
||||||
|
last_accessed_at=timezone.now() - timedelta(days=accessed_days_ago),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _make_rss_bridge(source_url, accessed_days_ago=0):
|
||||||
|
profile = Profile.objects.create(
|
||||||
|
feed=rss_feed_url(source_url),
|
||||||
|
title="Blog",
|
||||||
|
nick="blog",
|
||||||
|
version="v1",
|
||||||
|
)
|
||||||
|
return BridgedRssFeed.objects.create(
|
||||||
|
source_url=source_url,
|
||||||
|
profile=profile,
|
||||||
|
last_accessed_at=timezone.now() - timedelta(days=accessed_days_ago),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class RefreshBridgesTest(TestCase):
|
||||||
|
"""Test cases for the periodic bridge refresh."""
|
||||||
|
|
||||||
|
def test_only_recently_accessed_bridges_are_refreshed(self):
|
||||||
|
# Given: An active bridge and a dormant one
|
||||||
|
active = _make_ap_bridge("alice@m.example", accessed_days_ago=0)
|
||||||
|
_make_ap_bridge("bob@m.example", accessed_days_ago=BRIDGE_ACTIVE_DAYS + 1)
|
||||||
|
|
||||||
|
# When: The refresh task runs
|
||||||
|
with (
|
||||||
|
patch("app.bridge.store.refresh_activitypub_bridge") as mock_ap,
|
||||||
|
patch("app.bridge.store.refresh_rss_bridge"),
|
||||||
|
):
|
||||||
|
counters = _refresh_bridges_impl()
|
||||||
|
|
||||||
|
# Then: Only the active bridge is refreshed
|
||||||
|
self.assertEqual(counters, {"refreshed": 1, "failed": 0})
|
||||||
|
mock_ap.assert_called_once()
|
||||||
|
self.assertEqual(mock_ap.call_args[0][0].pk, active.pk)
|
||||||
|
|
||||||
|
def test_both_bridge_types_are_refreshed(self):
|
||||||
|
# Given: One active bridge of each type
|
||||||
|
_make_ap_bridge("alice@m.example")
|
||||||
|
_make_rss_bridge("https://blog.example/feed.xml")
|
||||||
|
|
||||||
|
# When: The refresh task runs
|
||||||
|
with (
|
||||||
|
patch("app.bridge.store.refresh_activitypub_bridge") as mock_ap,
|
||||||
|
patch("app.bridge.store.refresh_rss_bridge") as mock_rss,
|
||||||
|
):
|
||||||
|
counters = _refresh_bridges_impl()
|
||||||
|
|
||||||
|
# Then: Both are refreshed
|
||||||
|
self.assertEqual(counters, {"refreshed": 2, "failed": 0})
|
||||||
|
mock_ap.assert_called_once()
|
||||||
|
mock_rss.assert_called_once()
|
||||||
|
|
||||||
|
def test_failed_refreshes_are_counted_and_do_not_stop_the_task(self):
|
||||||
|
# Given: Two active bridges, the first origin failing
|
||||||
|
_make_ap_bridge("alice@m.example")
|
||||||
|
_make_ap_bridge("bob@m.example")
|
||||||
|
|
||||||
|
# When: The refresh task runs
|
||||||
|
with (
|
||||||
|
patch(
|
||||||
|
"app.bridge.store.refresh_activitypub_bridge",
|
||||||
|
side_effect=[BridgeError("down"), None],
|
||||||
|
),
|
||||||
|
patch("app.bridge.store.refresh_rss_bridge"),
|
||||||
|
):
|
||||||
|
counters = _refresh_bridges_impl()
|
||||||
|
|
||||||
|
# Then: The failure is counted and the other bridge still refreshed
|
||||||
|
self.assertEqual(counters, {"refreshed": 1, "failed": 1})
|
||||||
|
|
||||||
|
|
||||||
|
class CleanupStaleBridgesTest(TestCase):
|
||||||
|
"""Test cases for the stale bridge cleanup."""
|
||||||
|
|
||||||
|
def test_stale_bridges_are_deleted_with_profile_and_posts(self):
|
||||||
|
# Given: A stale bridge with posts and an active one
|
||||||
|
stale = _make_ap_bridge(
|
||||||
|
"old@m.example", accessed_days_ago=BRIDGE_STALE_DAYS + 1
|
||||||
|
)
|
||||||
|
Post.objects.create(
|
||||||
|
profile=stale.profile,
|
||||||
|
post_id="2025-05-01T10:00:00+00:00",
|
||||||
|
content="Bye",
|
||||||
|
created_at="2025-05-01T10:00:00+00:00",
|
||||||
|
)
|
||||||
|
active = _make_ap_bridge("new@m.example", accessed_days_ago=1)
|
||||||
|
|
||||||
|
# When: The cleanup task runs
|
||||||
|
deleted = _cleanup_stale_bridges_impl()
|
||||||
|
|
||||||
|
# Then: The stale bridge, its profile and posts are gone
|
||||||
|
self.assertEqual(deleted, 1)
|
||||||
|
self.assertFalse(BridgedActivityPubAccount.objects.filter(pk=stale.pk).exists())
|
||||||
|
self.assertFalse(
|
||||||
|
Profile.objects.filter(feed=activitypub_feed_url("old@m.example")).exists()
|
||||||
|
)
|
||||||
|
self.assertEqual(Post.objects.count(), 0)
|
||||||
|
# And: The active bridge survives
|
||||||
|
self.assertTrue(BridgedActivityPubAccount.objects.filter(pk=active.pk).exists())
|
||||||
|
|
||||||
|
def test_stale_rss_bridges_are_deleted_too(self):
|
||||||
|
# Given: A stale RSS bridge
|
||||||
|
_make_rss_bridge(
|
||||||
|
"https://blog.example/feed.xml",
|
||||||
|
accessed_days_ago=BRIDGE_STALE_DAYS + 1,
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The cleanup task runs
|
||||||
|
deleted = _cleanup_stale_bridges_impl()
|
||||||
|
|
||||||
|
# Then: It is deleted
|
||||||
|
self.assertEqual(deleted, 1)
|
||||||
|
self.assertEqual(BridgedRssFeed.objects.count(), 0)
|
||||||
|
|
||||||
|
def test_nothing_to_delete_returns_zero(self):
|
||||||
|
# Given: Only recently accessed bridges
|
||||||
|
_make_ap_bridge("alice@m.example", accessed_days_ago=1)
|
||||||
|
|
||||||
|
# When: The cleanup task runs
|
||||||
|
deleted = _cleanup_stale_bridges_impl()
|
||||||
|
|
||||||
|
# Then: Nothing is deleted
|
||||||
|
self.assertEqual(deleted, 0)
|
||||||
|
self.assertEqual(BridgedActivityPubAccount.objects.count(), 1)
|
||||||
383
app/bridge/test_views.py
Normal file
383
app/bridge/test_views.py
Normal file
|
|
@ -0,0 +1,383 @@
|
||||||
|
from datetime import timedelta
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from django.test import TestCase
|
||||||
|
from django.utils import timezone
|
||||||
|
|
||||||
|
from app.bridge.fetching import BridgeError
|
||||||
|
from app.bridge.models import (
|
||||||
|
BridgedActivityPubAccount,
|
||||||
|
BridgedRssFeed,
|
||||||
|
activitypub_feed_url,
|
||||||
|
rss_feed_url,
|
||||||
|
)
|
||||||
|
from app.feeds.models import Profile
|
||||||
|
|
||||||
|
AP_FEED_PATH = "/bridge/activitypub/@alice@m.example/"
|
||||||
|
RSS_SOURCE_URL = "https://blog.example/feed.xml"
|
||||||
|
|
||||||
|
|
||||||
|
def _ap_data(posts=None):
|
||||||
|
return {
|
||||||
|
"actor_url": "https://m.example/users/alice",
|
||||||
|
"outbox_url": "https://m.example/users/alice/outbox",
|
||||||
|
"metadata": {
|
||||||
|
"title": "Alice",
|
||||||
|
"nick": "alice",
|
||||||
|
"description": "Bio",
|
||||||
|
"avatar": "https://m.example/avatar.png",
|
||||||
|
"link": "https://m.example/@alice",
|
||||||
|
},
|
||||||
|
"posts": posts
|
||||||
|
if posts is not None
|
||||||
|
else [
|
||||||
|
{
|
||||||
|
"id": "2025-05-01T10:00:00+00:00",
|
||||||
|
"content": "Hello from the fediverse",
|
||||||
|
"tags": "",
|
||||||
|
"language": "",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _rss_data():
|
||||||
|
return {
|
||||||
|
"metadata": {
|
||||||
|
"title": "My Blog",
|
||||||
|
"nick": "My_Blog",
|
||||||
|
"description": "Notes",
|
||||||
|
"avatar": "",
|
||||||
|
"link": "https://blog.example",
|
||||||
|
"language": "en",
|
||||||
|
},
|
||||||
|
"posts": [
|
||||||
|
{
|
||||||
|
"id": "2025-05-01T10:00:00+00:00",
|
||||||
|
"content": "*** First article\n\nBody\n\n[[https://blog.example/first]]",
|
||||||
|
"tags": "",
|
||||||
|
"language": "",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class ActivityPubBridgeFeedViewTest(TestCase):
|
||||||
|
"""Test cases for the ActivityPub virtual feed endpoint."""
|
||||||
|
|
||||||
|
def test_first_get_registers_account_and_serves_org_file(self):
|
||||||
|
# Given: A reachable ActivityPub account not yet bridged
|
||||||
|
with patch(
|
||||||
|
"app.bridge.activitypub.fetch_account", return_value=_ap_data()
|
||||||
|
) as mock_fetch:
|
||||||
|
# When: The virtual feed is requested for the first time
|
||||||
|
response = self.client.get(AP_FEED_PATH)
|
||||||
|
|
||||||
|
# Then: The org file is served as plain text
|
||||||
|
self.assertEqual(response.status_code, 200)
|
||||||
|
self.assertTrue(response["Content-Type"].startswith("text/plain"))
|
||||||
|
content = response.content.decode()
|
||||||
|
self.assertIn("#+TITLE: Alice", content)
|
||||||
|
self.assertIn("#+NICK: alice", content)
|
||||||
|
self.assertIn("** 2025-05-01T10:00:00+00:00", content)
|
||||||
|
self.assertIn("Hello from the fediverse", content)
|
||||||
|
# And: The bridge and its profile are stored
|
||||||
|
mock_fetch.assert_called_once_with("alice", "m.example")
|
||||||
|
bridge = BridgedActivityPubAccount.objects.get(handle="alice@m.example")
|
||||||
|
self.assertEqual(bridge.profile.feed, activitypub_feed_url("alice@m.example"))
|
||||||
|
|
||||||
|
def test_second_get_is_served_from_database_without_network(self):
|
||||||
|
# Given: An already bridged account
|
||||||
|
with patch(
|
||||||
|
"app.bridge.activitypub.fetch_account", return_value=_ap_data()
|
||||||
|
) as mock_fetch:
|
||||||
|
self.client.get(AP_FEED_PATH)
|
||||||
|
|
||||||
|
# When: The feed is requested again shortly after
|
||||||
|
response = self.client.get(AP_FEED_PATH)
|
||||||
|
|
||||||
|
# Then: The response is correct and no second fetch happened
|
||||||
|
self.assertEqual(response.status_code, 200)
|
||||||
|
self.assertIn("Hello from the fediverse", response.content.decode())
|
||||||
|
self.assertEqual(mock_fetch.call_count, 1)
|
||||||
|
|
||||||
|
def test_non_canonical_handles_redirect_to_canonical_url(self):
|
||||||
|
# Given: Handle spellings with capitals or without the @ prefix
|
||||||
|
for path in (
|
||||||
|
"/bridge/activitypub/@Alice@M.Example/",
|
||||||
|
"/bridge/activitypub/alice@m.example/",
|
||||||
|
):
|
||||||
|
# When: The non-canonical URL is requested
|
||||||
|
response = self.client.get(path)
|
||||||
|
|
||||||
|
# Then: A permanent redirect points to the canonical URL
|
||||||
|
self.assertEqual(response.status_code, 301, path)
|
||||||
|
self.assertEqual(response["Location"], AP_FEED_PATH)
|
||||||
|
|
||||||
|
def test_invalid_handle_returns_400(self):
|
||||||
|
# Given: A handle without an instance part
|
||||||
|
# When: The feed is requested
|
||||||
|
response = self.client.get("/bridge/activitypub/@alice/")
|
||||||
|
|
||||||
|
# Then: The request is rejected
|
||||||
|
self.assertEqual(response.status_code, 400)
|
||||||
|
self.assertEqual(response.json()["type"], "Error")
|
||||||
|
|
||||||
|
def test_unreachable_account_returns_502(self):
|
||||||
|
# Given: An account whose instance cannot be reached
|
||||||
|
with patch(
|
||||||
|
"app.bridge.activitypub.fetch_account",
|
||||||
|
side_effect=BridgeError("connection refused"),
|
||||||
|
):
|
||||||
|
# When: The feed is requested
|
||||||
|
response = self.client.get(AP_FEED_PATH)
|
||||||
|
|
||||||
|
# Then: A bad gateway error with the envelope format is returned
|
||||||
|
self.assertEqual(response.status_code, 502)
|
||||||
|
body = response.json()
|
||||||
|
self.assertEqual(body["type"], "Error")
|
||||||
|
self.assertIsNone(body["data"])
|
||||||
|
# And: Nothing was stored
|
||||||
|
self.assertEqual(BridgedActivityPubAccount.objects.count(), 0)
|
||||||
|
|
||||||
|
def test_stale_bridge_is_refreshed_inline_on_access(self):
|
||||||
|
# Given: A bridged account whose data is older than a day
|
||||||
|
with patch("app.bridge.activitypub.fetch_account", return_value=_ap_data()):
|
||||||
|
self.client.get(AP_FEED_PATH)
|
||||||
|
bridge = BridgedActivityPubAccount.objects.get(handle="alice@m.example")
|
||||||
|
bridge.last_refreshed_at = timezone.now() - timedelta(days=2)
|
||||||
|
bridge.save()
|
||||||
|
|
||||||
|
new_posts = _ap_data(
|
||||||
|
posts=[
|
||||||
|
{
|
||||||
|
"id": "2025-05-05T10:00:00+00:00",
|
||||||
|
"content": "Fresh post",
|
||||||
|
"tags": "",
|
||||||
|
"language": "",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The feed is requested again
|
||||||
|
with patch(
|
||||||
|
"app.bridge.activitypub.fetch_account", return_value=new_posts
|
||||||
|
) as mock_fetch:
|
||||||
|
response = self.client.get(AP_FEED_PATH)
|
||||||
|
|
||||||
|
# Then: The origin was fetched again and the new post is served
|
||||||
|
mock_fetch.assert_called_once()
|
||||||
|
self.assertIn("Fresh post", response.content.decode())
|
||||||
|
bridge.refresh_from_db()
|
||||||
|
self.assertGreater(
|
||||||
|
bridge.last_refreshed_at, timezone.now() - timedelta(minutes=1)
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_failed_inline_refresh_serves_stored_copy(self):
|
||||||
|
# Given: A stale bridge whose origin is now unreachable
|
||||||
|
with patch("app.bridge.activitypub.fetch_account", return_value=_ap_data()):
|
||||||
|
self.client.get(AP_FEED_PATH)
|
||||||
|
bridge = BridgedActivityPubAccount.objects.get(handle="alice@m.example")
|
||||||
|
bridge.last_refreshed_at = timezone.now() - timedelta(days=2)
|
||||||
|
bridge.save()
|
||||||
|
|
||||||
|
# When: The feed is requested while the origin fails
|
||||||
|
with patch(
|
||||||
|
"app.bridge.activitypub.fetch_account",
|
||||||
|
side_effect=BridgeError("down"),
|
||||||
|
):
|
||||||
|
response = self.client.get(AP_FEED_PATH)
|
||||||
|
|
||||||
|
# Then: The stored copy is served anyway
|
||||||
|
self.assertEqual(response.status_code, 200)
|
||||||
|
self.assertIn("Hello from the fediverse", response.content.decode())
|
||||||
|
|
||||||
|
def test_bridged_profile_is_available_through_profile_endpoint(self):
|
||||||
|
# Given: A bridged account
|
||||||
|
with patch("app.bridge.activitypub.fetch_account", return_value=_ap_data()):
|
||||||
|
self.client.get(AP_FEED_PATH)
|
||||||
|
|
||||||
|
# When: The existing /profile/ endpoint is queried with the bridge URL
|
||||||
|
feed_url = activitypub_feed_url("alice@m.example")
|
||||||
|
response = self.client.get("/profile/", {"feed": feed_url})
|
||||||
|
|
||||||
|
# Then: The bridged profile is returned like any other profile
|
||||||
|
self.assertEqual(response.status_code, 200)
|
||||||
|
self.assertEqual(response.json()["type"], "Success")
|
||||||
|
|
||||||
|
|
||||||
|
class ActivityPubBridgeListViewTest(TestCase):
|
||||||
|
"""Test cases for the bridged accounts listing."""
|
||||||
|
|
||||||
|
def test_lists_bridged_accounts_with_feed_urls(self):
|
||||||
|
# Given: A bridged account
|
||||||
|
profile = Profile.objects.create(
|
||||||
|
feed=activitypub_feed_url("alice@m.example"),
|
||||||
|
title="Alice",
|
||||||
|
nick="alice",
|
||||||
|
version="v1",
|
||||||
|
)
|
||||||
|
BridgedActivityPubAccount.objects.create(
|
||||||
|
handle="alice@m.example",
|
||||||
|
actor_url="https://m.example/users/alice",
|
||||||
|
profile=profile,
|
||||||
|
last_accessed_at=timezone.now(),
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The list is requested
|
||||||
|
response = self.client.get("/bridge/activitypub/")
|
||||||
|
|
||||||
|
# Then: The account and its virtual feed URL are listed
|
||||||
|
self.assertEqual(response.status_code, 200)
|
||||||
|
body = response.json()
|
||||||
|
self.assertEqual(body["type"], "Success")
|
||||||
|
self.assertEqual(
|
||||||
|
body["data"],
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"handle": "@alice@m.example",
|
||||||
|
"feed": activitypub_feed_url("alice@m.example"),
|
||||||
|
}
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class RssBridgeViewTest(TestCase):
|
||||||
|
"""Test cases for the RSS bridge endpoint."""
|
||||||
|
|
||||||
|
def test_first_get_registers_feed_and_serves_org_file(self):
|
||||||
|
# Given: A reachable RSS feed not yet bridged
|
||||||
|
with patch(
|
||||||
|
"app.bridge.rss.fetch_rss_feed", return_value=_rss_data()
|
||||||
|
) as mock_fetch:
|
||||||
|
# When: The virtual feed is requested for the first time
|
||||||
|
response = self.client.get("/bridge/rss/", {"url": RSS_SOURCE_URL})
|
||||||
|
|
||||||
|
# Then: The org file is served as plain text
|
||||||
|
self.assertEqual(response.status_code, 200)
|
||||||
|
self.assertTrue(response["Content-Type"].startswith("text/plain"))
|
||||||
|
content = response.content.decode()
|
||||||
|
self.assertIn("#+TITLE: My Blog", content)
|
||||||
|
self.assertIn("#+NICK: My_Blog", content)
|
||||||
|
self.assertIn("*** First article", content)
|
||||||
|
# And: The bridge is stored
|
||||||
|
mock_fetch.assert_called_once_with(RSS_SOURCE_URL)
|
||||||
|
self.assertTrue(
|
||||||
|
BridgedRssFeed.objects.filter(source_url=RSS_SOURCE_URL).exists()
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_second_get_is_served_from_database_without_network(self):
|
||||||
|
# Given: An already bridged RSS feed
|
||||||
|
with patch(
|
||||||
|
"app.bridge.rss.fetch_rss_feed", return_value=_rss_data()
|
||||||
|
) as mock_fetch:
|
||||||
|
self.client.get("/bridge/rss/", {"url": RSS_SOURCE_URL})
|
||||||
|
|
||||||
|
# When: The feed is requested again shortly after
|
||||||
|
response = self.client.get("/bridge/rss/", {"url": RSS_SOURCE_URL})
|
||||||
|
|
||||||
|
# Then: The response is correct and no second fetch happened
|
||||||
|
self.assertEqual(response.status_code, 200)
|
||||||
|
self.assertEqual(mock_fetch.call_count, 1)
|
||||||
|
|
||||||
|
def test_without_url_parameter_lists_bridged_feeds(self):
|
||||||
|
# Given: A bridged RSS feed
|
||||||
|
profile = Profile.objects.create(
|
||||||
|
feed=rss_feed_url(RSS_SOURCE_URL),
|
||||||
|
title="My Blog",
|
||||||
|
nick="My_Blog",
|
||||||
|
version="v1",
|
||||||
|
)
|
||||||
|
BridgedRssFeed.objects.create(
|
||||||
|
source_url=RSS_SOURCE_URL,
|
||||||
|
profile=profile,
|
||||||
|
last_accessed_at=timezone.now(),
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The endpoint is requested without url
|
||||||
|
response = self.client.get("/bridge/rss/")
|
||||||
|
|
||||||
|
# Then: The bridged feeds are listed
|
||||||
|
self.assertEqual(response.status_code, 200)
|
||||||
|
body = response.json()
|
||||||
|
self.assertEqual(
|
||||||
|
body["data"],
|
||||||
|
[{"url": RSS_SOURCE_URL, "feed": rss_feed_url(RSS_SOURCE_URL)}],
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_invalid_url_parameter_returns_400(self):
|
||||||
|
# Given: Invalid url parameters
|
||||||
|
too_long = "https://example.com/" + "a" * 500
|
||||||
|
|
||||||
|
for bad_url in ("ftp://example.com/feed", "not-a-url", too_long):
|
||||||
|
# When: The endpoint is requested
|
||||||
|
response = self.client.get("/bridge/rss/", {"url": bad_url})
|
||||||
|
|
||||||
|
# Then: The request is rejected
|
||||||
|
self.assertEqual(response.status_code, 400, bad_url)
|
||||||
|
self.assertEqual(response.json()["type"], "Error")
|
||||||
|
|
||||||
|
def test_unreachable_feed_returns_502(self):
|
||||||
|
# Given: A feed URL that cannot be fetched
|
||||||
|
with patch(
|
||||||
|
"app.bridge.rss.fetch_rss_feed",
|
||||||
|
side_effect=BridgeError("timeout"),
|
||||||
|
):
|
||||||
|
# When: The endpoint is requested
|
||||||
|
response = self.client.get("/bridge/rss/", {"url": RSS_SOURCE_URL})
|
||||||
|
|
||||||
|
# Then: A bad gateway error is returned and nothing is stored
|
||||||
|
self.assertEqual(response.status_code, 502)
|
||||||
|
self.assertEqual(BridgedRssFeed.objects.count(), 0)
|
||||||
|
|
||||||
|
|
||||||
|
class BridgeIndexViewTest(TestCase):
|
||||||
|
"""Test cases for the bridge index endpoint."""
|
||||||
|
|
||||||
|
def test_index_exposes_bridge_links(self):
|
||||||
|
# When: The bridge index is requested
|
||||||
|
response = self.client.get("/bridge/")
|
||||||
|
|
||||||
|
# Then: Both bridge types are discoverable
|
||||||
|
self.assertEqual(response.status_code, 200)
|
||||||
|
links = response.json()["_links"]
|
||||||
|
self.assertIn("activitypub-feed", links)
|
||||||
|
self.assertIn("rss-feed", links)
|
||||||
|
|
||||||
|
def test_root_endpoint_links_to_bridges(self):
|
||||||
|
# When: The relay root is requested
|
||||||
|
response = self.client.get("/")
|
||||||
|
|
||||||
|
# Then: The bridge endpoints are listed
|
||||||
|
links = response.json()["_links"]
|
||||||
|
self.assertIn("bridge", links)
|
||||||
|
self.assertIn("bridge-activitypub", links)
|
||||||
|
self.assertIn("bridge-rss", links)
|
||||||
|
|
||||||
|
|
||||||
|
class LastAccessedTrackingTest(TestCase):
|
||||||
|
"""Test cases for access tracking used by refresh and cleanup."""
|
||||||
|
|
||||||
|
def test_old_last_accessed_is_updated_on_get(self):
|
||||||
|
# Given: A bridge last accessed two days ago
|
||||||
|
profile = Profile.objects.create(
|
||||||
|
feed=activitypub_feed_url("alice@m.example"),
|
||||||
|
title="Alice",
|
||||||
|
nick="alice",
|
||||||
|
version="v1",
|
||||||
|
)
|
||||||
|
old_access = timezone.now() - timedelta(days=2)
|
||||||
|
bridge = BridgedActivityPubAccount.objects.create(
|
||||||
|
handle="alice@m.example",
|
||||||
|
actor_url="https://m.example/users/alice",
|
||||||
|
profile=profile,
|
||||||
|
last_accessed_at=old_access,
|
||||||
|
last_refreshed_at=timezone.now(),
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The feed is requested
|
||||||
|
self.client.get(AP_FEED_PATH)
|
||||||
|
|
||||||
|
# Then: last_accessed_at moves forward
|
||||||
|
bridge.refresh_from_db()
|
||||||
|
self.assertGreater(bridge.last_accessed_at, old_access)
|
||||||
23
app/bridge/urls.py
Normal file
23
app/bridge/urls.py
Normal file
|
|
@ -0,0 +1,23 @@
|
||||||
|
from django.urls import path, re_path
|
||||||
|
|
||||||
|
from .views import (
|
||||||
|
ActivityPubBridgeFeedView,
|
||||||
|
ActivityPubBridgeListView,
|
||||||
|
BridgeIndexView,
|
||||||
|
RssBridgeView,
|
||||||
|
)
|
||||||
|
|
||||||
|
urlpatterns = [
|
||||||
|
path("", BridgeIndexView.as_view(), name="bridge-index"),
|
||||||
|
path(
|
||||||
|
"activitypub/",
|
||||||
|
ActivityPubBridgeListView.as_view(),
|
||||||
|
name="bridge-activitypub-list",
|
||||||
|
),
|
||||||
|
re_path(
|
||||||
|
r"^activitypub/(?P<handle>[^/]+)/$",
|
||||||
|
ActivityPubBridgeFeedView.as_view(),
|
||||||
|
name="bridge-activitypub-feed",
|
||||||
|
),
|
||||||
|
path("rss/", RssBridgeView.as_view(), name="bridge-rss"),
|
||||||
|
]
|
||||||
207
app/bridge/views.py
Normal file
207
app/bridge/views.py
Normal file
|
|
@ -0,0 +1,207 @@
|
||||||
|
"""
|
||||||
|
Bridge endpoints: expose ActivityPub accounts and RSS/Atom feeds as
|
||||||
|
virtual social.org files that any Org Social client can #+FOLLOW:.
|
||||||
|
|
||||||
|
Registration is implicit: the first GET of an unknown account fetches
|
||||||
|
it from the origin and stores it. Later requests are served from the
|
||||||
|
database; the periodic refresh task keeps active bridges up to date,
|
||||||
|
and a request older than REFRESH_ON_ACCESS triggers an inline refresh.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from datetime import timedelta
|
||||||
|
|
||||||
|
from django.http import HttpResponse, HttpResponsePermanentRedirect
|
||||||
|
from django.utils import timezone
|
||||||
|
from rest_framework import status
|
||||||
|
from rest_framework.response import Response
|
||||||
|
from rest_framework.views import APIView
|
||||||
|
|
||||||
|
from . import store
|
||||||
|
from .activitypub import normalize_handle
|
||||||
|
from .fetching import BridgeError
|
||||||
|
from .models import BridgedActivityPubAccount, BridgedRssFeed
|
||||||
|
from .org_renderer import render_profile_org
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# A GET on data older than this triggers a synchronous refresh, which
|
||||||
|
# also reactivates bridges that fell out of the periodic refresh window
|
||||||
|
REFRESH_ON_ACCESS = timedelta(hours=24)
|
||||||
|
# last_accessed_at is written at most once per hour
|
||||||
|
ACCESS_UPDATE_THROTTLE = timedelta(hours=1)
|
||||||
|
|
||||||
|
MAX_SOURCE_URL_LENGTH = 500
|
||||||
|
|
||||||
|
|
||||||
|
def _error_response(errors, http_status):
|
||||||
|
return Response(
|
||||||
|
{"type": "Error", "errors": errors, "data": None}, status=http_status
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _serve_bridge(bridge, refresh_callable):
|
||||||
|
"""Common flow: refresh if stale, track access, render the org file."""
|
||||||
|
now = timezone.now()
|
||||||
|
|
||||||
|
if (
|
||||||
|
bridge.last_refreshed_at is None
|
||||||
|
or now - bridge.last_refreshed_at > REFRESH_ON_ACCESS
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
refresh_callable(bridge)
|
||||||
|
except BridgeError as e:
|
||||||
|
# Serve the stored copy; the origin may recover later
|
||||||
|
logger.warning(f"Inline refresh failed for {bridge}: {e}")
|
||||||
|
|
||||||
|
if now - bridge.last_accessed_at > ACCESS_UPDATE_THROTTLE:
|
||||||
|
bridge.last_accessed_at = now
|
||||||
|
bridge.save(update_fields=["last_accessed_at"])
|
||||||
|
|
||||||
|
posts = bridge.profile.posts.order_by("created_at")
|
||||||
|
content = render_profile_org(bridge.profile, posts)
|
||||||
|
return HttpResponse(content, content_type="text/plain; charset=utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
class BridgeIndexView(APIView):
|
||||||
|
"""Information about the available bridges."""
|
||||||
|
|
||||||
|
def get(self, request):
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Success",
|
||||||
|
"errors": [],
|
||||||
|
"data": {
|
||||||
|
"description": (
|
||||||
|
"Bridges expose external accounts as virtual "
|
||||||
|
"social.org feeds that can be followed with #+FOLLOW:"
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"_links": {
|
||||||
|
"self": {"href": "/bridge/", "method": "GET"},
|
||||||
|
"activitypub-feed": {
|
||||||
|
"href": "/bridge/activitypub/@{user}@{instance}/",
|
||||||
|
"method": "GET",
|
||||||
|
"templated": True,
|
||||||
|
},
|
||||||
|
"activitypub-list": {
|
||||||
|
"href": "/bridge/activitypub/",
|
||||||
|
"method": "GET",
|
||||||
|
},
|
||||||
|
"rss-feed": {
|
||||||
|
"href": "/bridge/rss/?url={feed_url}",
|
||||||
|
"method": "GET",
|
||||||
|
"templated": True,
|
||||||
|
},
|
||||||
|
"rss-list": {"href": "/bridge/rss/", "method": "GET"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ActivityPubBridgeListView(APIView):
|
||||||
|
"""List bridged ActivityPub accounts."""
|
||||||
|
|
||||||
|
def get(self, request):
|
||||||
|
data = [
|
||||||
|
{"handle": f"@{bridge.handle}", "feed": bridge.feed_url}
|
||||||
|
for bridge in BridgedActivityPubAccount.objects.all()
|
||||||
|
]
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Success",
|
||||||
|
"errors": [],
|
||||||
|
"data": data,
|
||||||
|
"_links": {
|
||||||
|
"self": {"href": "/bridge/activitypub/", "method": "GET"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ActivityPubBridgeFeedView(APIView):
|
||||||
|
"""Serve an ActivityPub account as a virtual social.org file."""
|
||||||
|
|
||||||
|
def get(self, request, handle):
|
||||||
|
normalized = normalize_handle(handle)
|
||||||
|
if normalized is None:
|
||||||
|
return _error_response(
|
||||||
|
["Invalid handle. Expected format: @user@instance"],
|
||||||
|
status.HTTP_400_BAD_REQUEST,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Redirect non-canonical spellings so each account has one URL
|
||||||
|
if handle != f"@{normalized}":
|
||||||
|
return HttpResponsePermanentRedirect(f"/bridge/activitypub/@{normalized}/")
|
||||||
|
|
||||||
|
bridge = (
|
||||||
|
BridgedActivityPubAccount.objects.select_related("profile")
|
||||||
|
.filter(handle=normalized)
|
||||||
|
.first()
|
||||||
|
)
|
||||||
|
|
||||||
|
if bridge is None:
|
||||||
|
try:
|
||||||
|
bridge = store.create_activitypub_bridge(normalized)
|
||||||
|
except BridgeError as e:
|
||||||
|
logger.warning(f"Could not bridge @{normalized}: {e}")
|
||||||
|
return _error_response(
|
||||||
|
[f"Could not fetch ActivityPub account: {e}"],
|
||||||
|
status.HTTP_502_BAD_GATEWAY,
|
||||||
|
)
|
||||||
|
|
||||||
|
return _serve_bridge(bridge, store.refresh_activitypub_bridge)
|
||||||
|
|
||||||
|
|
||||||
|
class RssBridgeView(APIView):
|
||||||
|
"""
|
||||||
|
With ?url= serves an RSS/Atom feed as a virtual social.org file;
|
||||||
|
without it, lists the bridged RSS feeds.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def get(self, request):
|
||||||
|
source_url = request.query_params.get("url")
|
||||||
|
|
||||||
|
if source_url is None:
|
||||||
|
data = [
|
||||||
|
{"url": bridge.source_url, "feed": bridge.feed_url}
|
||||||
|
for bridge in BridgedRssFeed.objects.all()
|
||||||
|
]
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Success",
|
||||||
|
"errors": [],
|
||||||
|
"data": data,
|
||||||
|
"_links": {
|
||||||
|
"self": {"href": "/bridge/rss/", "method": "GET"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
source_url = source_url.strip()
|
||||||
|
if (
|
||||||
|
not source_url.startswith(("http://", "https://"))
|
||||||
|
or len(source_url) > MAX_SOURCE_URL_LENGTH
|
||||||
|
):
|
||||||
|
return _error_response(
|
||||||
|
["Invalid url parameter. Expected an http(s) feed URL"],
|
||||||
|
status.HTTP_400_BAD_REQUEST,
|
||||||
|
)
|
||||||
|
|
||||||
|
bridge = (
|
||||||
|
BridgedRssFeed.objects.select_related("profile")
|
||||||
|
.filter(source_url=source_url)
|
||||||
|
.first()
|
||||||
|
)
|
||||||
|
|
||||||
|
if bridge is None:
|
||||||
|
try:
|
||||||
|
bridge = store.create_rss_bridge(source_url)
|
||||||
|
except BridgeError as e:
|
||||||
|
logger.warning(f"Could not bridge RSS feed {source_url}: {e}")
|
||||||
|
return _error_response(
|
||||||
|
[f"Could not fetch RSS feed: {e}"],
|
||||||
|
status.HTTP_502_BAD_GATEWAY,
|
||||||
|
)
|
||||||
|
|
||||||
|
return _serve_bridge(bridge, store.refresh_rss_bridge)
|
||||||
0
app/feedcontent/__init__.py
Normal file
0
app/feedcontent/__init__.py
Normal file
6
app/feedcontent/apps.py
Normal file
6
app/feedcontent/apps.py
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
from django.apps import AppConfig
|
||||||
|
|
||||||
|
|
||||||
|
class FeedcontentConfig(AppConfig):
|
||||||
|
default_auto_field = "django.db.models.BigAutoField"
|
||||||
|
name = "app.feedcontent"
|
||||||
204
app/feedcontent/tests.py
Normal file
204
app/feedcontent/tests.py
Normal file
|
|
@ -0,0 +1,204 @@
|
||||||
|
from django.test import TestCase
|
||||||
|
from rest_framework.test import APIClient
|
||||||
|
from unittest.mock import patch, Mock
|
||||||
|
from app.feeds.models import Profile
|
||||||
|
import requests
|
||||||
|
|
||||||
|
|
||||||
|
class FeedContentViewTests(TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
"""Set up test fixtures."""
|
||||||
|
self.client = APIClient()
|
||||||
|
self.profile = Profile.objects.create(
|
||||||
|
feed="https://example.com/social.org",
|
||||||
|
title="Test Profile",
|
||||||
|
nick="testuser",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_get_feed_content_success(self):
|
||||||
|
"""Test successfully fetching feed content"""
|
||||||
|
# Given: A registered feed whose server returns valid Org Social content
|
||||||
|
mock_content = """#+TITLE: My Social Feed
|
||||||
|
#+AUTHOR: John Doe
|
||||||
|
|
||||||
|
* 2025-02-05T10:00:00+0100
|
||||||
|
:PROPERTIES:
|
||||||
|
:ID: 2025-02-05T10:00:00+0100
|
||||||
|
:END:
|
||||||
|
|
||||||
|
Hello, world! This is my first post.
|
||||||
|
"""
|
||||||
|
|
||||||
|
with patch("app.feedcontent.views.requests.get") as mock_get:
|
||||||
|
mock_response = Mock()
|
||||||
|
mock_response.status_code = 200
|
||||||
|
mock_response.content = mock_content.encode("utf-8")
|
||||||
|
mock_response.raise_for_status = Mock()
|
||||||
|
mock_get.return_value = mock_response
|
||||||
|
|
||||||
|
# When: The feed content endpoint is requested
|
||||||
|
response = self.client.get(
|
||||||
|
"/feed-content/", {"feed": "https://example.com/social.org"}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: The content is returned verbatim with HATEOAS links
|
||||||
|
self.assertEqual(response.status_code, 200)
|
||||||
|
self.assertEqual(response.data["type"], "Success")
|
||||||
|
self.assertEqual(response.data["data"]["content"], mock_content)
|
||||||
|
self.assertIn("_links", response.data)
|
||||||
|
self.assertIn("self", response.data["_links"])
|
||||||
|
|
||||||
|
def test_get_feed_content_missing_parameter(self):
|
||||||
|
"""Test error when feed parameter is missing"""
|
||||||
|
# Given: No feed parameter
|
||||||
|
|
||||||
|
# When: The feed content endpoint is requested
|
||||||
|
response = self.client.get("/feed-content/")
|
||||||
|
|
||||||
|
# Then: A 400 error explains the missing parameter
|
||||||
|
self.assertEqual(response.status_code, 400)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn("Feed URL parameter is required", response.data["errors"])
|
||||||
|
|
||||||
|
def test_get_feed_content_empty_parameter(self):
|
||||||
|
"""Test error when feed parameter is empty"""
|
||||||
|
# Given: A blank feed parameter
|
||||||
|
|
||||||
|
# When: The feed content endpoint is requested
|
||||||
|
response = self.client.get("/feed-content/", {"feed": " "})
|
||||||
|
|
||||||
|
# Then: A 400 error explains the missing parameter
|
||||||
|
self.assertEqual(response.status_code, 400)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn("Feed URL parameter is required", response.data["errors"])
|
||||||
|
|
||||||
|
def test_get_feed_content_feed_not_found(self):
|
||||||
|
"""Test error when feed is not registered in relay"""
|
||||||
|
# Given: A feed URL that is not registered in the relay
|
||||||
|
|
||||||
|
# When: The feed content endpoint is requested
|
||||||
|
response = self.client.get(
|
||||||
|
"/feed-content/", {"feed": "https://unknown.com/social.org"}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: A 404 error explains the feed is unknown
|
||||||
|
self.assertEqual(response.status_code, 404)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn("Feed not found in relay", response.data["errors"])
|
||||||
|
|
||||||
|
def test_get_feed_content_timeout(self):
|
||||||
|
"""Test error when feed server times out"""
|
||||||
|
# Given: A registered feed whose server times out
|
||||||
|
with patch("app.feedcontent.views.requests.get") as mock_get:
|
||||||
|
mock_get.side_effect = requests.exceptions.Timeout()
|
||||||
|
|
||||||
|
# When: The feed content endpoint is requested
|
||||||
|
response = self.client.get(
|
||||||
|
"/feed-content/", {"feed": "https://example.com/social.org"}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: A 502 error reports the timeout
|
||||||
|
self.assertEqual(response.status_code, 502)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn("Request timeout", response.data["errors"][0])
|
||||||
|
|
||||||
|
def test_get_feed_content_connection_error(self):
|
||||||
|
"""Test error when connection to feed server fails"""
|
||||||
|
# Given: A registered feed whose server refuses the connection
|
||||||
|
with patch("app.feedcontent.views.requests.get") as mock_get:
|
||||||
|
mock_get.side_effect = requests.exceptions.ConnectionError()
|
||||||
|
|
||||||
|
# When: The feed content endpoint is requested
|
||||||
|
response = self.client.get(
|
||||||
|
"/feed-content/", {"feed": "https://example.com/social.org"}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: A 502 error reports the connection failure
|
||||||
|
self.assertEqual(response.status_code, 502)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn("Connection error", response.data["errors"][0])
|
||||||
|
|
||||||
|
def test_get_feed_content_http_error(self):
|
||||||
|
"""Test error when feed server returns HTTP error"""
|
||||||
|
# Given: A registered feed whose server answers with an HTTP error
|
||||||
|
with patch("app.feedcontent.views.requests.get") as mock_get:
|
||||||
|
mock_response = Mock()
|
||||||
|
mock_response.raise_for_status.side_effect = requests.exceptions.HTTPError(
|
||||||
|
"404 Not Found"
|
||||||
|
)
|
||||||
|
mock_get.return_value = mock_response
|
||||||
|
|
||||||
|
# When: The feed content endpoint is requested
|
||||||
|
response = self.client.get(
|
||||||
|
"/feed-content/", {"feed": "https://example.com/social.org"}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: A 502 error reports the upstream HTTP error
|
||||||
|
self.assertEqual(response.status_code, 502)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn("HTTP error", response.data["errors"][0])
|
||||||
|
|
||||||
|
def test_get_feed_content_unicode(self):
|
||||||
|
"""Test fetching feed content with unicode characters"""
|
||||||
|
# Given: A registered feed whose content includes unicode and emojis
|
||||||
|
mock_content = """#+TITLE: Mi Feed Social
|
||||||
|
#+AUTHOR: José García
|
||||||
|
|
||||||
|
* 2025-02-05T10:00:00+0100
|
||||||
|
:PROPERTIES:
|
||||||
|
:ID: 2025-02-05T10:00:00+0100
|
||||||
|
:END:
|
||||||
|
|
||||||
|
¡Hola mundo! Este es mi primer post con émojis 🚀 ❤️
|
||||||
|
"""
|
||||||
|
|
||||||
|
with patch("app.feedcontent.views.requests.get") as mock_get:
|
||||||
|
mock_response = Mock()
|
||||||
|
mock_response.status_code = 200
|
||||||
|
mock_response.content = mock_content.encode("utf-8")
|
||||||
|
mock_response.raise_for_status = Mock()
|
||||||
|
mock_get.return_value = mock_response
|
||||||
|
|
||||||
|
# When: The feed content endpoint is requested
|
||||||
|
response = self.client.get(
|
||||||
|
"/feed-content/", {"feed": "https://example.com/social.org"}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: The unicode content survives the round trip intact
|
||||||
|
self.assertEqual(response.status_code, 200)
|
||||||
|
self.assertEqual(response.data["type"], "Success")
|
||||||
|
self.assertEqual(response.data["data"]["content"], mock_content)
|
||||||
|
self.assertIn("🚀", response.data["data"]["content"])
|
||||||
|
self.assertIn("❤️", response.data["data"]["content"])
|
||||||
|
|
||||||
|
def test_get_feed_content_preserves_formatting(self):
|
||||||
|
"""Test that feed content preserves whitespace and formatting"""
|
||||||
|
# Given: A registered feed whose content mixes blank lines, spaces and tabs
|
||||||
|
mock_content = """#+TITLE: Test Feed
|
||||||
|
|
||||||
|
* 2025-02-05T10:00:00+0100
|
||||||
|
:PROPERTIES:
|
||||||
|
:ID: 2025-02-05T10:00:00+0100
|
||||||
|
:END:
|
||||||
|
|
||||||
|
Line 1
|
||||||
|
|
||||||
|
Line 2 with multiple spaces
|
||||||
|
Line 3 with tab
|
||||||
|
"""
|
||||||
|
|
||||||
|
with patch("app.feedcontent.views.requests.get") as mock_get:
|
||||||
|
mock_response = Mock()
|
||||||
|
mock_response.status_code = 200
|
||||||
|
mock_response.content = mock_content.encode("utf-8")
|
||||||
|
mock_response.raise_for_status = Mock()
|
||||||
|
mock_get.return_value = mock_response
|
||||||
|
|
||||||
|
# When: The feed content endpoint is requested
|
||||||
|
response = self.client.get(
|
||||||
|
"/feed-content/", {"feed": "https://example.com/social.org"}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: The content is exactly as provided, preserving all whitespace
|
||||||
|
self.assertEqual(response.status_code, 200)
|
||||||
|
self.assertEqual(response.data["data"]["content"], mock_content)
|
||||||
6
app/feedcontent/urls.py
Normal file
6
app/feedcontent/urls.py
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
from django.urls import path
|
||||||
|
from .views import FeedContentView
|
||||||
|
|
||||||
|
urlpatterns = [
|
||||||
|
path("", FeedContentView.as_view(), name="feed-content"),
|
||||||
|
]
|
||||||
106
app/feedcontent/views.py
Normal file
106
app/feedcontent/views.py
Normal file
|
|
@ -0,0 +1,106 @@
|
||||||
|
from rest_framework.views import APIView
|
||||||
|
from rest_framework.response import Response
|
||||||
|
from rest_framework import status
|
||||||
|
import logging
|
||||||
|
import requests
|
||||||
|
|
||||||
|
from app.feeds.models import Profile
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class FeedContentView(APIView):
|
||||||
|
"""Get raw content of an Org Social feed file"""
|
||||||
|
|
||||||
|
def get(self, request):
|
||||||
|
feed_url = request.query_params.get("feed")
|
||||||
|
|
||||||
|
if not feed_url or not feed_url.strip():
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Error",
|
||||||
|
"errors": ["Feed URL parameter is required"],
|
||||||
|
"data": None,
|
||||||
|
},
|
||||||
|
status=status.HTTP_400_BAD_REQUEST,
|
||||||
|
)
|
||||||
|
|
||||||
|
feed_url = feed_url.strip()
|
||||||
|
|
||||||
|
# Check if the feed is registered in the relay
|
||||||
|
try:
|
||||||
|
Profile.objects.get(feed=feed_url)
|
||||||
|
except Profile.DoesNotExist:
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Error",
|
||||||
|
"errors": ["Feed not found in relay"],
|
||||||
|
"data": None,
|
||||||
|
},
|
||||||
|
status=status.HTTP_404_NOT_FOUND,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Fetch the feed content from the source URL
|
||||||
|
try:
|
||||||
|
response = requests.get(feed_url, timeout=10)
|
||||||
|
response.raise_for_status()
|
||||||
|
|
||||||
|
# Decode content as UTF-8
|
||||||
|
content = response.content.decode("utf-8")
|
||||||
|
|
||||||
|
# URL encode the feed_url for the self link
|
||||||
|
from urllib.parse import quote
|
||||||
|
|
||||||
|
encoded_feed_url = quote(feed_url, safe="")
|
||||||
|
|
||||||
|
response_data = {
|
||||||
|
"type": "Success",
|
||||||
|
"errors": [],
|
||||||
|
"data": {"content": content},
|
||||||
|
"_links": {
|
||||||
|
"self": {
|
||||||
|
"href": f"/feed-content/?feed={encoded_feed_url}",
|
||||||
|
"method": "GET",
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
return Response(response_data, status=status.HTTP_200_OK)
|
||||||
|
|
||||||
|
except requests.exceptions.Timeout:
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Error",
|
||||||
|
"errors": ["Request timeout: Feed server did not respond in time"],
|
||||||
|
"data": None,
|
||||||
|
},
|
||||||
|
status=status.HTTP_502_BAD_GATEWAY,
|
||||||
|
)
|
||||||
|
except requests.exceptions.ConnectionError:
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Error",
|
||||||
|
"errors": ["Connection error: Could not reach feed server"],
|
||||||
|
"data": None,
|
||||||
|
},
|
||||||
|
status=status.HTTP_502_BAD_GATEWAY,
|
||||||
|
)
|
||||||
|
except requests.exceptions.HTTPError as e:
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Error",
|
||||||
|
"errors": [f"HTTP error: {e}"],
|
||||||
|
"data": None,
|
||||||
|
},
|
||||||
|
status=status.HTTP_502_BAD_GATEWAY,
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to fetch feed content from {feed_url}: {e}")
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Error",
|
||||||
|
"errors": ["Failed to fetch feed content"],
|
||||||
|
"data": None,
|
||||||
|
},
|
||||||
|
status=status.HTTP_502_BAD_GATEWAY,
|
||||||
|
)
|
||||||
49
app/feeds/cors_middleware.py
Normal file
49
app/feeds/cors_middleware.py
Normal file
|
|
@ -0,0 +1,49 @@
|
||||||
|
"""
|
||||||
|
Middleware for adding CORS headers to all responses.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class CORSMiddleware:
|
||||||
|
"""
|
||||||
|
Middleware that adds CORS headers to all responses.
|
||||||
|
|
||||||
|
This middleware ensures that:
|
||||||
|
- All API responses are accessible from any origin (Access-Control-Allow-Origin: *)
|
||||||
|
- Preflight OPTIONS requests are handled correctly
|
||||||
|
- Common HTTP methods and headers are allowed
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, get_response):
|
||||||
|
self.get_response = get_response
|
||||||
|
|
||||||
|
def __call__(self, request):
|
||||||
|
# Handle preflight OPTIONS requests
|
||||||
|
if request.method == "OPTIONS":
|
||||||
|
response = self._create_options_response()
|
||||||
|
else:
|
||||||
|
# Get the response from the view
|
||||||
|
response = self.get_response(request)
|
||||||
|
|
||||||
|
# Add CORS headers to all responses
|
||||||
|
response["Access-Control-Allow-Origin"] = "*"
|
||||||
|
response["Access-Control-Allow-Methods"] = (
|
||||||
|
"GET, POST, PUT, PATCH, DELETE, OPTIONS"
|
||||||
|
)
|
||||||
|
response["Access-Control-Allow-Headers"] = (
|
||||||
|
"Content-Type, Authorization, X-Requested-With"
|
||||||
|
)
|
||||||
|
response["Access-Control-Max-Age"] = "86400" # 24 hours
|
||||||
|
|
||||||
|
return response
|
||||||
|
|
||||||
|
def _create_options_response(self):
|
||||||
|
"""Create a response for OPTIONS preflight requests"""
|
||||||
|
from django.http import HttpResponse
|
||||||
|
|
||||||
|
response = HttpResponse()
|
||||||
|
response.status_code = 200
|
||||||
|
return response
|
||||||
21
app/feeds/migrations/0008_post_include.py
Normal file
21
app/feeds/migrations/0008_post_include.py
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
# Generated by Django 5.2.8 on 2025-11-16 17:34
|
||||||
|
|
||||||
|
from django.db import migrations, models
|
||||||
|
|
||||||
|
|
||||||
|
class Migration(migrations.Migration):
|
||||||
|
dependencies = [
|
||||||
|
("feeds", "0007_relaymetadata"),
|
||||||
|
]
|
||||||
|
|
||||||
|
operations = [
|
||||||
|
migrations.AddField(
|
||||||
|
model_name="post",
|
||||||
|
name="include",
|
||||||
|
field=models.CharField(
|
||||||
|
blank=True,
|
||||||
|
help_text="Post being boosted/shared (URL#ID format)",
|
||||||
|
max_length=300,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
]
|
||||||
19
app/feeds/migrations/0009_alter_post_created_at.py
Normal file
19
app/feeds/migrations/0009_alter_post_created_at.py
Normal file
|
|
@ -0,0 +1,19 @@
|
||||||
|
# Generated by Django 5.2.6 on 2025-12-25 09:01
|
||||||
|
|
||||||
|
import django.utils.timezone
|
||||||
|
from django.db import migrations, models
|
||||||
|
|
||||||
|
|
||||||
|
class Migration(migrations.Migration):
|
||||||
|
|
||||||
|
dependencies = [
|
||||||
|
('feeds', '0008_post_include'),
|
||||||
|
]
|
||||||
|
|
||||||
|
operations = [
|
||||||
|
migrations.AlterField(
|
||||||
|
model_name='post',
|
||||||
|
name='created_at',
|
||||||
|
field=models.DateTimeField(default=django.utils.timezone.now, help_text='Creation timestamp, parsed from post_id when available'),
|
||||||
|
),
|
||||||
|
]
|
||||||
|
|
@ -0,0 +1,33 @@
|
||||||
|
# Generated by Django 5.2.6 on 2026-01-05 12:49
|
||||||
|
|
||||||
|
from django.db import migrations, models
|
||||||
|
|
||||||
|
|
||||||
|
class Migration(migrations.Migration):
|
||||||
|
|
||||||
|
dependencies = [
|
||||||
|
('feeds', '0009_alter_post_created_at'),
|
||||||
|
]
|
||||||
|
|
||||||
|
operations = [
|
||||||
|
migrations.AddField(
|
||||||
|
model_name='profile',
|
||||||
|
name='birthday',
|
||||||
|
field=models.DateField(blank=True, help_text='User birthday in YYYY-MM-DD format', null=True),
|
||||||
|
),
|
||||||
|
migrations.AddField(
|
||||||
|
model_name='profile',
|
||||||
|
name='language',
|
||||||
|
field=models.CharField(blank=True, help_text='Space-separated language codes (ISO 639-1)', max_length=100),
|
||||||
|
),
|
||||||
|
migrations.AddField(
|
||||||
|
model_name='profile',
|
||||||
|
name='location',
|
||||||
|
field=models.CharField(blank=True, help_text='User location (city, country, etc.)', max_length=200),
|
||||||
|
),
|
||||||
|
migrations.AddField(
|
||||||
|
model_name='profile',
|
||||||
|
name='pinned',
|
||||||
|
field=models.CharField(blank=True, help_text='Pinned post ID (timestamp)', max_length=50),
|
||||||
|
),
|
||||||
|
]
|
||||||
31
app/feeds/migrations/0011_outgoingwebmention.py
Normal file
31
app/feeds/migrations/0011_outgoingwebmention.py
Normal file
|
|
@ -0,0 +1,31 @@
|
||||||
|
# Generated by Django 6.0.7 on 2026-07-16 08:13
|
||||||
|
|
||||||
|
from django.db import migrations, models
|
||||||
|
|
||||||
|
|
||||||
|
class Migration(migrations.Migration):
|
||||||
|
|
||||||
|
dependencies = [
|
||||||
|
('feeds', '0010_profile_birthday_profile_language_profile_location_and_more'),
|
||||||
|
]
|
||||||
|
|
||||||
|
operations = [
|
||||||
|
migrations.CreateModel(
|
||||||
|
name='OutgoingWebmention',
|
||||||
|
fields=[
|
||||||
|
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
|
||||||
|
('source', models.URLField(help_text='Post URL (feed#post_id) containing the link', max_length=500)),
|
||||||
|
('target', models.URLField(help_text='External URL the post links to', max_length=500)),
|
||||||
|
('endpoint', models.URLField(blank=True, help_text='Discovered Webmention endpoint', max_length=500)),
|
||||||
|
('status', models.CharField(choices=[('pending', 'Pending'), ('sent', 'Sent'), ('failed', 'Failed'), ('no_endpoint', 'No endpoint')], default='pending', max_length=20)),
|
||||||
|
('attempts', models.PositiveSmallIntegerField(default=0)),
|
||||||
|
('response_code', models.IntegerField(blank=True, help_text='HTTP status of the last delivery attempt', null=True)),
|
||||||
|
('last_attempt_at', models.DateTimeField(blank=True, null=True)),
|
||||||
|
('created_at', models.DateTimeField(auto_now_add=True)),
|
||||||
|
],
|
||||||
|
options={
|
||||||
|
'ordering': ['created_at'],
|
||||||
|
'unique_together': {('source', 'target')},
|
||||||
|
},
|
||||||
|
),
|
||||||
|
]
|
||||||
24
app/feeds/migrations/0012_delete_bridge_feeds.py
Normal file
24
app/feeds/migrations/0012_delete_bridge_feeds.py
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
from django.db import migrations
|
||||||
|
from django.db.models import Q
|
||||||
|
|
||||||
|
|
||||||
|
def delete_bridge_feeds(apps, schema_editor):
|
||||||
|
"""
|
||||||
|
Bridge virtual feeds are a connection helper, not real accounts.
|
||||||
|
Remove any that were registered before they were excluded from
|
||||||
|
registration and discovery.
|
||||||
|
"""
|
||||||
|
Feed = apps.get_model("feeds", "Feed")
|
||||||
|
Feed.objects.filter(
|
||||||
|
Q(url__contains="/bridge/activitypub/") | Q(url__contains="/bridge/rss/")
|
||||||
|
).delete()
|
||||||
|
|
||||||
|
|
||||||
|
class Migration(migrations.Migration):
|
||||||
|
dependencies = [
|
||||||
|
("feeds", "0011_outgoingwebmention"),
|
||||||
|
]
|
||||||
|
|
||||||
|
operations = [
|
||||||
|
migrations.RunPython(delete_bridge_feeds, migrations.RunPython.noop),
|
||||||
|
]
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
from django.db import models
|
from django.db import models
|
||||||
|
from django.utils import timezone
|
||||||
import logging
|
import logging
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
@ -18,6 +19,20 @@ class Profile(models.Model):
|
||||||
avatar = models.URLField(
|
avatar = models.URLField(
|
||||||
blank=True, help_text="URL to avatar image (128x128px JPG/PNG)"
|
blank=True, help_text="URL to avatar image (128x128px JPG/PNG)"
|
||||||
)
|
)
|
||||||
|
location = models.CharField(
|
||||||
|
max_length=200, blank=True, help_text="User location (city, country, etc.)"
|
||||||
|
)
|
||||||
|
birthday = models.DateField(
|
||||||
|
blank=True, null=True, help_text="User birthday in YYYY-MM-DD format"
|
||||||
|
)
|
||||||
|
language = models.CharField(
|
||||||
|
max_length=100,
|
||||||
|
blank=True,
|
||||||
|
help_text="Space-separated language codes (ISO 639-1)",
|
||||||
|
)
|
||||||
|
pinned = models.CharField(
|
||||||
|
max_length=50, blank=True, help_text="Pinned post ID (timestamp)"
|
||||||
|
)
|
||||||
version = models.CharField(
|
version = models.CharField(
|
||||||
max_length=50,
|
max_length=50,
|
||||||
blank=True,
|
blank=True,
|
||||||
|
|
@ -121,6 +136,11 @@ class Post(models.Model):
|
||||||
group = models.CharField(
|
group = models.CharField(
|
||||||
max_length=100, blank=True, help_text="Group name (GROUP property)"
|
max_length=100, blank=True, help_text="Group name (GROUP property)"
|
||||||
)
|
)
|
||||||
|
include = models.CharField(
|
||||||
|
max_length=300,
|
||||||
|
blank=True,
|
||||||
|
help_text="Post being boosted/shared (URL#ID format)",
|
||||||
|
)
|
||||||
|
|
||||||
# Poll related fields
|
# Poll related fields
|
||||||
poll_end = models.DateTimeField(
|
poll_end = models.DateTimeField(
|
||||||
|
|
@ -128,7 +148,10 @@ class Post(models.Model):
|
||||||
)
|
)
|
||||||
|
|
||||||
# Metadata
|
# Metadata
|
||||||
created_at = models.DateTimeField(auto_now_add=True)
|
created_at = models.DateTimeField(
|
||||||
|
default=timezone.now,
|
||||||
|
help_text="Creation timestamp, parsed from post_id when available",
|
||||||
|
)
|
||||||
updated_at = models.DateTimeField(auto_now=True)
|
updated_at = models.DateTimeField(auto_now=True)
|
||||||
|
|
||||||
class Meta:
|
class Meta:
|
||||||
|
|
@ -146,6 +169,10 @@ class Post(models.Model):
|
||||||
def is_reply(self):
|
def is_reply(self):
|
||||||
return bool(self.reply_to)
|
return bool(self.reply_to)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def is_boost(self):
|
||||||
|
return bool(self.include)
|
||||||
|
|
||||||
|
|
||||||
class PollOption(models.Model):
|
class PollOption(models.Model):
|
||||||
"""
|
"""
|
||||||
|
|
@ -230,6 +257,51 @@ class Feed(models.Model):
|
||||||
return self.url
|
return self.url
|
||||||
|
|
||||||
|
|
||||||
|
class OutgoingWebmention(models.Model):
|
||||||
|
"""
|
||||||
|
Outbox of Webmentions sent by the relay on behalf of feed authors.
|
||||||
|
|
||||||
|
The (source, target) pair is unique, which guarantees each link found
|
||||||
|
in a post is notified at most once no matter how many times the feed
|
||||||
|
is rescanned.
|
||||||
|
"""
|
||||||
|
|
||||||
|
STATUS_PENDING = "pending"
|
||||||
|
STATUS_SENT = "sent"
|
||||||
|
STATUS_FAILED = "failed"
|
||||||
|
STATUS_NO_ENDPOINT = "no_endpoint"
|
||||||
|
STATUS_CHOICES = [
|
||||||
|
(STATUS_PENDING, "Pending"),
|
||||||
|
(STATUS_SENT, "Sent"),
|
||||||
|
(STATUS_FAILED, "Failed"),
|
||||||
|
(STATUS_NO_ENDPOINT, "No endpoint"),
|
||||||
|
]
|
||||||
|
|
||||||
|
source = models.URLField(
|
||||||
|
max_length=500, help_text="Post URL (feed#post_id) containing the link"
|
||||||
|
)
|
||||||
|
target = models.URLField(max_length=500, help_text="External URL the post links to")
|
||||||
|
endpoint = models.URLField(
|
||||||
|
max_length=500, blank=True, help_text="Discovered Webmention endpoint"
|
||||||
|
)
|
||||||
|
status = models.CharField(
|
||||||
|
max_length=20, choices=STATUS_CHOICES, default=STATUS_PENDING
|
||||||
|
)
|
||||||
|
attempts = models.PositiveSmallIntegerField(default=0)
|
||||||
|
response_code = models.IntegerField(
|
||||||
|
null=True, blank=True, help_text="HTTP status of the last delivery attempt"
|
||||||
|
)
|
||||||
|
last_attempt_at = models.DateTimeField(null=True, blank=True)
|
||||||
|
created_at = models.DateTimeField(auto_now_add=True)
|
||||||
|
|
||||||
|
class Meta:
|
||||||
|
unique_together = ["source", "target"]
|
||||||
|
ordering = ["created_at"]
|
||||||
|
|
||||||
|
def __str__(self):
|
||||||
|
return f"{self.source} -> {self.target} ({self.status})"
|
||||||
|
|
||||||
|
|
||||||
class RelayMetadata(models.Model):
|
class RelayMetadata(models.Model):
|
||||||
"""
|
"""
|
||||||
Global metadata for the relay - used for HTTP caching headers.
|
Global metadata for the relay - used for HTTP caching headers.
|
||||||
|
|
|
||||||
61
app/feeds/notification_publisher.py
Normal file
61
app/feeds/notification_publisher.py
Normal file
|
|
@ -0,0 +1,61 @@
|
||||||
|
"""
|
||||||
|
Notification publisher for real-time SSE notifications.
|
||||||
|
|
||||||
|
This module handles publishing notifications to Redis Pub/Sub channels
|
||||||
|
so that SSE clients can receive real-time updates.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import redis
|
||||||
|
from django.conf import settings
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def get_redis_connection():
|
||||||
|
"""Get a Redis connection using Huey settings."""
|
||||||
|
try:
|
||||||
|
redis_host = settings.HUEY["connection"]["host"]
|
||||||
|
redis_port = settings.HUEY["connection"]["port"]
|
||||||
|
redis_db = settings.HUEY["connection"]["db"]
|
||||||
|
|
||||||
|
return redis.Redis(
|
||||||
|
host=redis_host, port=redis_port, db=redis_db, decode_responses=True
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to create Redis connection: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def publish_notification(target_feed_url, notification_type, post_url, **extra_data):
|
||||||
|
"""
|
||||||
|
Publish a notification to a feed's Redis Pub/Sub channel.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
target_feed_url: URL of the feed that should receive the notification
|
||||||
|
notification_type: Type of notification ("mention", "reply", "reaction", "boost")
|
||||||
|
post_url: Full post URL (feed_url#post_id) that triggered the notification
|
||||||
|
**extra_data: Additional data to include in the notification (e.g., emoji, parent)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
bool: True if published successfully, False otherwise
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
r = get_redis_connection()
|
||||||
|
if not r:
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Build notification payload
|
||||||
|
notification = {"type": notification_type, "post": post_url, **extra_data}
|
||||||
|
|
||||||
|
# Publish to the target feed's channel
|
||||||
|
channel = f"notifications:{target_feed_url}"
|
||||||
|
r.publish(channel, json.dumps(notification))
|
||||||
|
|
||||||
|
logger.debug(f"Published {notification_type} notification to {target_feed_url}")
|
||||||
|
return True
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to publish notification: {e}")
|
||||||
|
return False
|
||||||
|
|
@ -1,7 +1,37 @@
|
||||||
import re
|
import re
|
||||||
import requests
|
import requests
|
||||||
|
from datetime import datetime
|
||||||
from typing import Dict, Any, Tuple
|
from typing import Dict, Any, Tuple
|
||||||
from django.utils import timezone
|
from django.utils import timezone
|
||||||
|
import logging
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# (connect, read) timeouts in seconds for fetching remote feeds.
|
||||||
|
# A short connect timeout drops unreachable hosts quickly (e.g. dead servers),
|
||||||
|
# while the read timeout stays generous enough for slow-but-alive servers.
|
||||||
|
FEED_FETCH_TIMEOUT = (3.05, 5)
|
||||||
|
|
||||||
|
|
||||||
|
def _sanitize_birthday(value: str) -> str:
|
||||||
|
"""Return the birthday only if it is a valid YYYY-MM-DD date, else "".
|
||||||
|
|
||||||
|
Some feeds use other date formats (e.g. "2003/06/17"). The Profile.birthday
|
||||||
|
DateField only accepts YYYY-MM-DD, so an invalid value would raise a
|
||||||
|
ValidationError and abort the whole feed scan. We drop the malformed value
|
||||||
|
instead of discarding an otherwise valid feed.
|
||||||
|
"""
|
||||||
|
value = (value or "").strip()
|
||||||
|
if not value:
|
||||||
|
return ""
|
||||||
|
try:
|
||||||
|
datetime.strptime(value, "%Y-%m-%d")
|
||||||
|
except ValueError:
|
||||||
|
logger.warning(
|
||||||
|
f"Ignoring invalid birthday value (expected YYYY-MM-DD): {value!r}"
|
||||||
|
)
|
||||||
|
return ""
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
def _update_feed_last_successful_fetch(url: str):
|
def _update_feed_last_successful_fetch(url: str):
|
||||||
|
|
@ -19,6 +49,211 @@ def _update_feed_last_successful_fetch(url: str):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _handle_feed_redirect(old_url: str, new_url: str):
|
||||||
|
"""
|
||||||
|
Handle feed URL redirect by updating or merging feeds.
|
||||||
|
|
||||||
|
If a feed redirects to a new URL:
|
||||||
|
1. Check if new URL already exists as a feed
|
||||||
|
2. If yes: merge data and delete old URL
|
||||||
|
3. If no: update old URL to new URL
|
||||||
|
|
||||||
|
Args:
|
||||||
|
old_url: Original URL that redirected
|
||||||
|
new_url: Final URL after redirect
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from .models import Feed, Profile, Post, Follow, Mention, PollVote
|
||||||
|
from django.db import transaction
|
||||||
|
|
||||||
|
old_feed = Feed.objects.filter(url=old_url).first()
|
||||||
|
new_feed = Feed.objects.filter(url=new_url).first()
|
||||||
|
|
||||||
|
if old_feed and new_feed:
|
||||||
|
# Both URLs exist - merge them
|
||||||
|
logger.info(f"Feed redirect detected: {old_url} -> {new_url}")
|
||||||
|
logger.info("Both feeds exist. Merging old feed into new feed.")
|
||||||
|
|
||||||
|
with transaction.atomic():
|
||||||
|
# Get profiles for both feeds
|
||||||
|
old_profile = Profile.objects.filter(feed=old_url).first()
|
||||||
|
new_profile = Profile.objects.filter(feed=new_url).first()
|
||||||
|
|
||||||
|
if old_profile and new_profile:
|
||||||
|
# Merge profiles - keep the new one, migrate relationships
|
||||||
|
logger.info(
|
||||||
|
f"Merging profile data: {old_profile.nick} -> {new_profile.nick}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Migrate Follow relationships where old_profile is followed
|
||||||
|
follows_as_followed = Follow.objects.filter(followed=old_profile)
|
||||||
|
for follow in follows_as_followed:
|
||||||
|
# Check if this relationship already exists with new_profile
|
||||||
|
existing = Follow.objects.filter(
|
||||||
|
follower=follow.follower, followed=new_profile
|
||||||
|
).first()
|
||||||
|
|
||||||
|
if not existing:
|
||||||
|
# Update to point to new_profile
|
||||||
|
follow.followed = new_profile
|
||||||
|
try:
|
||||||
|
follow.save()
|
||||||
|
logger.debug(
|
||||||
|
f"Migrated follow relationship: {follow.follower.nick} -> {new_profile.nick}"
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(
|
||||||
|
f"Could not migrate follow relationship, deleting: {e}"
|
||||||
|
)
|
||||||
|
follow.delete()
|
||||||
|
else:
|
||||||
|
# Relationship already exists, delete duplicate
|
||||||
|
follow.delete()
|
||||||
|
logger.debug("Deleted duplicate follow relationship")
|
||||||
|
|
||||||
|
# Migrate Follow relationships where old_profile is follower
|
||||||
|
follows_as_follower = Follow.objects.filter(follower=old_profile)
|
||||||
|
for follow in follows_as_follower:
|
||||||
|
# Check if this relationship already exists with new_profile
|
||||||
|
existing = Follow.objects.filter(
|
||||||
|
follower=new_profile, followed=follow.followed
|
||||||
|
).first()
|
||||||
|
|
||||||
|
if not existing:
|
||||||
|
# Update to point to new_profile
|
||||||
|
follow.follower = new_profile
|
||||||
|
try:
|
||||||
|
follow.save()
|
||||||
|
logger.debug(
|
||||||
|
f"Migrated follow relationship: {new_profile.nick} -> {follow.followed.nick}"
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(
|
||||||
|
f"Could not migrate follow relationship, deleting: {e}"
|
||||||
|
)
|
||||||
|
follow.delete()
|
||||||
|
else:
|
||||||
|
# Relationship already exists, delete duplicate
|
||||||
|
follow.delete()
|
||||||
|
logger.debug("Deleted duplicate follow relationship")
|
||||||
|
|
||||||
|
# Migrate Mention relationships pointing to old_profile
|
||||||
|
# Mentions have unique constraint (post, mentioned_profile)
|
||||||
|
mentions_to_migrate = Mention.objects.filter(
|
||||||
|
mentioned_profile=old_profile
|
||||||
|
)
|
||||||
|
for mention in mentions_to_migrate:
|
||||||
|
# Check if this mention already exists with new_profile
|
||||||
|
existing = Mention.objects.filter(
|
||||||
|
post=mention.post, mentioned_profile=new_profile
|
||||||
|
).first()
|
||||||
|
|
||||||
|
if not existing:
|
||||||
|
# Update to point to new_profile
|
||||||
|
mention.mentioned_profile = new_profile
|
||||||
|
try:
|
||||||
|
mention.save()
|
||||||
|
logger.debug(
|
||||||
|
f"Migrated mention in post {mention.post.post_id}"
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(
|
||||||
|
f"Could not migrate mention, deleting: {e}"
|
||||||
|
)
|
||||||
|
mention.delete()
|
||||||
|
else:
|
||||||
|
# Mention already exists, delete duplicate
|
||||||
|
mention.delete()
|
||||||
|
logger.debug("Deleted duplicate mention")
|
||||||
|
|
||||||
|
# Migrate posts from old_profile to new_profile (avoid duplicates)
|
||||||
|
old_posts = Post.objects.filter(profile=old_profile)
|
||||||
|
for old_post in old_posts:
|
||||||
|
# Check if post already exists in new profile
|
||||||
|
existing_post = Post.objects.filter(
|
||||||
|
profile=new_profile, post_id=old_post.post_id
|
||||||
|
).first()
|
||||||
|
|
||||||
|
if not existing_post:
|
||||||
|
# Migrate post to new profile
|
||||||
|
old_post.profile = new_profile
|
||||||
|
old_post.save()
|
||||||
|
logger.debug(
|
||||||
|
f"Migrated post {old_post.post_id} to new profile"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Post already exists, handle poll votes carefully
|
||||||
|
# Get all poll votes pointing to old_post
|
||||||
|
poll_votes = PollVote.objects.filter(poll_post=old_post)
|
||||||
|
for poll_vote in poll_votes:
|
||||||
|
try:
|
||||||
|
# Check if this vote already exists for the existing_post
|
||||||
|
existing_vote = PollVote.objects.filter(
|
||||||
|
post=poll_vote.post, poll_post=existing_post
|
||||||
|
).first()
|
||||||
|
|
||||||
|
if not existing_vote:
|
||||||
|
# Update to point to existing_post
|
||||||
|
poll_vote.poll_post = existing_post
|
||||||
|
poll_vote.save()
|
||||||
|
logger.debug(
|
||||||
|
"Migrated poll vote to existing post"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Vote already exists, delete duplicate
|
||||||
|
poll_vote.delete()
|
||||||
|
logger.debug("Deleted duplicate poll vote")
|
||||||
|
except Exception as e:
|
||||||
|
# If there's any constraint error, just delete the vote
|
||||||
|
logger.warning(
|
||||||
|
f"Error migrating poll vote, deleting: {e}"
|
||||||
|
)
|
||||||
|
poll_vote.delete()
|
||||||
|
|
||||||
|
# Delete duplicate post
|
||||||
|
old_post.delete()
|
||||||
|
logger.debug(f"Removed duplicate post {old_post.post_id}")
|
||||||
|
|
||||||
|
# Delete old profile
|
||||||
|
old_profile.delete()
|
||||||
|
logger.info(f"Deleted old profile: {old_url}")
|
||||||
|
|
||||||
|
elif old_profile and not new_profile:
|
||||||
|
# Only old profile exists - update its feed URL
|
||||||
|
logger.info(f"Updating profile feed URL: {old_url} -> {new_url}")
|
||||||
|
old_profile.feed = new_url
|
||||||
|
old_profile.save()
|
||||||
|
|
||||||
|
# Delete the old feed
|
||||||
|
old_feed.delete()
|
||||||
|
logger.info(f"Deleted old feed: {old_url}")
|
||||||
|
|
||||||
|
elif old_feed and not new_feed:
|
||||||
|
# Only old URL exists - update it to new URL
|
||||||
|
logger.info(f"Feed redirect detected: {old_url} -> {new_url}")
|
||||||
|
logger.info(f"Updating feed URL to: {new_url}")
|
||||||
|
|
||||||
|
with transaction.atomic():
|
||||||
|
# Update the feed URL
|
||||||
|
old_feed.url = new_url
|
||||||
|
old_feed.save()
|
||||||
|
|
||||||
|
# Update all profiles pointing to old URL
|
||||||
|
profiles_updated = Profile.objects.filter(feed=old_url).update(
|
||||||
|
feed=new_url
|
||||||
|
)
|
||||||
|
logger.info(f"Updated {profiles_updated} profile(s) to new URL")
|
||||||
|
|
||||||
|
# If new_feed exists but not old_feed, nothing to do
|
||||||
|
# This can happen if the redirect was already processed
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
# Don't break parsing if redirect handling fails
|
||||||
|
logger.error(
|
||||||
|
f"Failed to handle redirect {old_url} -> {new_url}: {e}", exc_info=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def parse_org_social(url: str) -> Dict[str, Any]:
|
def parse_org_social(url: str) -> Dict[str, Any]:
|
||||||
"""
|
"""
|
||||||
Parse an Org Social file from a URL and return structured data.
|
Parse an Org Social file from a URL and return structured data.
|
||||||
|
|
@ -30,11 +265,22 @@ def parse_org_social(url: str) -> Dict[str, Any]:
|
||||||
Dictionary containing parsed metadata and posts
|
Dictionary containing parsed metadata and posts
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
response = requests.get(url, timeout=5)
|
response = requests.get(url, timeout=FEED_FETCH_TIMEOUT)
|
||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
# Decode content as UTF-8 explicitly to avoid encoding issues
|
# Decode content as UTF-8 explicitly to avoid encoding issues
|
||||||
# when the server doesn't specify charset in Content-Type header
|
# when the server doesn't specify charset in Content-Type header
|
||||||
content = response.content.decode('utf-8')
|
content = response.content.decode("utf-8")
|
||||||
|
|
||||||
|
# Check if URL was redirected
|
||||||
|
final_url = response.url
|
||||||
|
if final_url != url and response.history:
|
||||||
|
# URL was redirected - handle the redirect
|
||||||
|
logger.info(
|
||||||
|
f"Redirect detected: {url} -> {final_url} (status: {response.history[0].status_code})"
|
||||||
|
)
|
||||||
|
_handle_feed_redirect(url, final_url)
|
||||||
|
# Use final URL for further operations
|
||||||
|
url = final_url
|
||||||
|
|
||||||
# Update last_successful_fetch if we got a 200 response
|
# Update last_successful_fetch if we got a 200 response
|
||||||
if response.status_code == 200:
|
if response.status_code == 200:
|
||||||
|
|
@ -50,6 +296,10 @@ def parse_org_social(url: str) -> Dict[str, Any]:
|
||||||
"nick": "",
|
"nick": "",
|
||||||
"description": "",
|
"description": "",
|
||||||
"avatar": "",
|
"avatar": "",
|
||||||
|
"location": "",
|
||||||
|
"birthday": "",
|
||||||
|
"language": "",
|
||||||
|
"pinned": "",
|
||||||
"links": [],
|
"links": [],
|
||||||
"follows": [],
|
"follows": [],
|
||||||
"contacts": [],
|
"contacts": [],
|
||||||
|
|
@ -78,6 +328,25 @@ def parse_org_social(url: str) -> Dict[str, Any]:
|
||||||
avatar_match = re.search(r"^\s*\#\+AVATAR:\s*(.+)$", content, re.MULTILINE)
|
avatar_match = re.search(r"^\s*\#\+AVATAR:\s*(.+)$", content, re.MULTILINE)
|
||||||
result["metadata"]["avatar"] = avatar_match.group(1).strip() if avatar_match else ""
|
result["metadata"]["avatar"] = avatar_match.group(1).strip() if avatar_match else ""
|
||||||
|
|
||||||
|
# Parse new v1.6 fields
|
||||||
|
location_match = re.search(r"^\s*\#\+LOCATION:\s*(.+)$", content, re.MULTILINE)
|
||||||
|
result["metadata"]["location"] = (
|
||||||
|
location_match.group(1).strip() if location_match else ""
|
||||||
|
)
|
||||||
|
|
||||||
|
birthday_match = re.search(r"^\s*\#\+BIRTHDAY:\s*(.+)$", content, re.MULTILINE)
|
||||||
|
result["metadata"]["birthday"] = (
|
||||||
|
_sanitize_birthday(birthday_match.group(1)) if birthday_match else ""
|
||||||
|
)
|
||||||
|
|
||||||
|
language_match = re.search(r"^\s*\#\+LANGUAGE:\s*(.+)$", content, re.MULTILINE)
|
||||||
|
result["metadata"]["language"] = (
|
||||||
|
language_match.group(1).strip() if language_match else ""
|
||||||
|
)
|
||||||
|
|
||||||
|
pinned_match = re.search(r"^\s*\#\+PINNED:\s*(.+)$", content, re.MULTILINE)
|
||||||
|
result["metadata"]["pinned"] = pinned_match.group(1).strip() if pinned_match else ""
|
||||||
|
|
||||||
# Parse multiple values
|
# Parse multiple values
|
||||||
result["metadata"]["links"] = [
|
result["metadata"]["links"] = [
|
||||||
match.group(1).strip()
|
match.group(1).strip()
|
||||||
|
|
@ -109,14 +378,18 @@ def parse_org_social(url: str) -> Dict[str, Any]:
|
||||||
# Split posts by ** headers (exactly 2 asterisks, not 3+)
|
# Split posts by ** headers (exactly 2 asterisks, not 3+)
|
||||||
# Use negative lookahead (?!\*) to ensure we don't match *** or ****
|
# Use negative lookahead (?!\*) to ensure we don't match *** or ****
|
||||||
# Use ^ anchor to match ** only at start of line
|
# Use ^ anchor to match ** only at start of line
|
||||||
post_pattern = r"^\*\*(?!\*)[^\n]*\n(?::PROPERTIES:\s*\n((?::[^:\n]+:[^\n]*\n)*):END:\s*\n)?(.*?)(?=^\*\*(?!\*)|\Z)"
|
# Capture group 1: header content (can contain ID in v1.6)
|
||||||
|
# Capture group 2: properties text
|
||||||
|
# Capture group 3: post content
|
||||||
|
post_pattern = r"^\*\*(?!\*)([^\n]*)\n(?::PROPERTIES:\s*\n((?::[^:\n]+:[^\n]*\n)*):END:\s*\n)?(.*?)(?=^\*\*(?!\*)|\Z)"
|
||||||
post_matches = re.finditer(
|
post_matches = re.finditer(
|
||||||
post_pattern, posts_content, re.DOTALL | re.MULTILINE
|
post_pattern, posts_content, re.DOTALL | re.MULTILINE
|
||||||
)
|
)
|
||||||
|
|
||||||
for post_match in post_matches:
|
for post_match in post_matches:
|
||||||
properties_text = post_match.group(1) or ""
|
header_text = post_match.group(1).strip() if post_match.group(1) else ""
|
||||||
content_text = post_match.group(2).strip() if post_match.group(2) else ""
|
properties_text = post_match.group(2) or ""
|
||||||
|
content_text = post_match.group(3).strip() if post_match.group(3) else ""
|
||||||
|
|
||||||
post: Dict[str, Any] = {
|
post: Dict[str, Any] = {
|
||||||
"id": "",
|
"id": "",
|
||||||
|
|
@ -126,6 +399,17 @@ def parse_org_social(url: str) -> Dict[str, Any]:
|
||||||
"poll_options": [],
|
"poll_options": [],
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# First check if ID is in header (v1.6 feature)
|
||||||
|
# Header ID takes priority over property drawer ID
|
||||||
|
if header_text:
|
||||||
|
# RFC 3339 format: ####-##-##T##:##:##[+-]####
|
||||||
|
header_id_match = re.match(
|
||||||
|
r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{2}:?\d{2}$",
|
||||||
|
header_text,
|
||||||
|
)
|
||||||
|
if header_id_match:
|
||||||
|
post["id"] = header_text
|
||||||
|
|
||||||
# Parse properties
|
# Parse properties
|
||||||
if properties_text:
|
if properties_text:
|
||||||
# Use [ \t]* instead of \s* to avoid capturing newlines
|
# Use [ \t]* instead of \s* to avoid capturing newlines
|
||||||
|
|
@ -136,7 +420,8 @@ def parse_org_social(url: str) -> Dict[str, Any]:
|
||||||
# Only add non-empty properties
|
# Only add non-empty properties
|
||||||
if prop_value:
|
if prop_value:
|
||||||
post["properties"][prop_name] = prop_value
|
post["properties"][prop_name] = prop_value
|
||||||
if prop_name == "id":
|
# If ID not already set from header, use property ID
|
||||||
|
if prop_name == "id" and not post["id"]:
|
||||||
post["id"] = prop_value
|
post["id"] = prop_value
|
||||||
|
|
||||||
# Extract mentions from content
|
# Extract mentions from content
|
||||||
|
|
@ -176,6 +461,10 @@ def parse_org_social_content(content: str) -> Dict[str, Any]:
|
||||||
"nick": "",
|
"nick": "",
|
||||||
"description": "",
|
"description": "",
|
||||||
"avatar": "",
|
"avatar": "",
|
||||||
|
"location": "",
|
||||||
|
"birthday": "",
|
||||||
|
"language": "",
|
||||||
|
"pinned": "",
|
||||||
"links": [],
|
"links": [],
|
||||||
"follows": [],
|
"follows": [],
|
||||||
"contacts": [],
|
"contacts": [],
|
||||||
|
|
@ -204,6 +493,25 @@ def parse_org_social_content(content: str) -> Dict[str, Any]:
|
||||||
avatar_match = re.search(r"^\s*\#\+AVATAR:\s*(.+)$", content, re.MULTILINE)
|
avatar_match = re.search(r"^\s*\#\+AVATAR:\s*(.+)$", content, re.MULTILINE)
|
||||||
result["metadata"]["avatar"] = avatar_match.group(1).strip() if avatar_match else ""
|
result["metadata"]["avatar"] = avatar_match.group(1).strip() if avatar_match else ""
|
||||||
|
|
||||||
|
# Parse new v1.6 fields
|
||||||
|
location_match = re.search(r"^\s*\#\+LOCATION:\s*(.+)$", content, re.MULTILINE)
|
||||||
|
result["metadata"]["location"] = (
|
||||||
|
location_match.group(1).strip() if location_match else ""
|
||||||
|
)
|
||||||
|
|
||||||
|
birthday_match = re.search(r"^\s*\#\+BIRTHDAY:\s*(.+)$", content, re.MULTILINE)
|
||||||
|
result["metadata"]["birthday"] = (
|
||||||
|
_sanitize_birthday(birthday_match.group(1)) if birthday_match else ""
|
||||||
|
)
|
||||||
|
|
||||||
|
language_match = re.search(r"^\s*\#\+LANGUAGE:\s*(.+)$", content, re.MULTILINE)
|
||||||
|
result["metadata"]["language"] = (
|
||||||
|
language_match.group(1).strip() if language_match else ""
|
||||||
|
)
|
||||||
|
|
||||||
|
pinned_match = re.search(r"^\s*\#\+PINNED:\s*(.+)$", content, re.MULTILINE)
|
||||||
|
result["metadata"]["pinned"] = pinned_match.group(1).strip() if pinned_match else ""
|
||||||
|
|
||||||
# Parse multiple values
|
# Parse multiple values
|
||||||
result["metadata"]["links"] = [
|
result["metadata"]["links"] = [
|
||||||
match.group(1).strip()
|
match.group(1).strip()
|
||||||
|
|
@ -235,14 +543,18 @@ def parse_org_social_content(content: str) -> Dict[str, Any]:
|
||||||
# Split posts by ** headers (exactly 2 asterisks, not 3+)
|
# Split posts by ** headers (exactly 2 asterisks, not 3+)
|
||||||
# Use negative lookahead (?!\*) to ensure we don't match *** or ****
|
# Use negative lookahead (?!\*) to ensure we don't match *** or ****
|
||||||
# Use ^ anchor to match ** only at start of line
|
# Use ^ anchor to match ** only at start of line
|
||||||
post_pattern = r"^\*\*(?!\*)[^\n]*\n(?::PROPERTIES:\s*\n((?::[^:\n]+:[^\n]*\n)*):END:\s*\n)?(.*?)(?=^\*\*(?!\*)|\Z)"
|
# Capture group 1: header content (can contain ID in v1.6)
|
||||||
|
# Capture group 2: properties text
|
||||||
|
# Capture group 3: post content
|
||||||
|
post_pattern = r"^\*\*(?!\*)([^\n]*)\n(?::PROPERTIES:\s*\n((?::[^:\n]+:[^\n]*\n)*):END:\s*\n)?(.*?)(?=^\*\*(?!\*)|\Z)"
|
||||||
post_matches = re.finditer(
|
post_matches = re.finditer(
|
||||||
post_pattern, posts_content, re.DOTALL | re.MULTILINE
|
post_pattern, posts_content, re.DOTALL | re.MULTILINE
|
||||||
)
|
)
|
||||||
|
|
||||||
for post_match in post_matches:
|
for post_match in post_matches:
|
||||||
properties_text = post_match.group(1) or ""
|
header_text = post_match.group(1).strip() if post_match.group(1) else ""
|
||||||
content_text = post_match.group(2).strip() if post_match.group(2) else ""
|
properties_text = post_match.group(2) or ""
|
||||||
|
content_text = post_match.group(3).strip() if post_match.group(3) else ""
|
||||||
|
|
||||||
post: Dict[str, Any] = {
|
post: Dict[str, Any] = {
|
||||||
"id": "",
|
"id": "",
|
||||||
|
|
@ -252,6 +564,17 @@ def parse_org_social_content(content: str) -> Dict[str, Any]:
|
||||||
"poll_options": [],
|
"poll_options": [],
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# First check if ID is in header (v1.6 feature)
|
||||||
|
# Header ID takes priority over property drawer ID
|
||||||
|
if header_text:
|
||||||
|
# RFC 3339 format: ####-##-##T##:##:##[+-]####
|
||||||
|
header_id_match = re.match(
|
||||||
|
r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{2}:?\d{2}$",
|
||||||
|
header_text,
|
||||||
|
)
|
||||||
|
if header_id_match:
|
||||||
|
post["id"] = header_text
|
||||||
|
|
||||||
# Parse properties
|
# Parse properties
|
||||||
if properties_text:
|
if properties_text:
|
||||||
# Use [ \t]* instead of \s* to avoid capturing newlines
|
# Use [ \t]* instead of \s* to avoid capturing newlines
|
||||||
|
|
@ -262,7 +585,8 @@ def parse_org_social_content(content: str) -> Dict[str, Any]:
|
||||||
# Only add non-empty properties
|
# Only add non-empty properties
|
||||||
if prop_value:
|
if prop_value:
|
||||||
post["properties"][prop_name] = prop_value
|
post["properties"][prop_name] = prop_value
|
||||||
if prop_name == "id":
|
# If ID not already set from header, use property ID
|
||||||
|
if prop_name == "id" and not post["id"]:
|
||||||
post["id"] = prop_value
|
post["id"] = prop_value
|
||||||
|
|
||||||
# Extract mentions from content
|
# Extract mentions from content
|
||||||
|
|
@ -297,15 +621,23 @@ def validate_org_social_feed(url: str) -> Tuple[bool, str]:
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
# Check if URL responds with 200
|
# Check if URL responds with 200
|
||||||
response = requests.get(url, timeout=5)
|
response = requests.get(url, timeout=FEED_FETCH_TIMEOUT)
|
||||||
if response.status_code != 200:
|
if response.status_code != 200:
|
||||||
return False, f"URL returned status code {response.status_code}"
|
return False, f"URL returned status code {response.status_code}"
|
||||||
|
|
||||||
|
# Check if URL was redirected
|
||||||
|
final_url = response.url
|
||||||
|
if final_url != url and response.history:
|
||||||
|
logger.info(f"Validation: Redirect detected: {url} -> {final_url}")
|
||||||
|
_handle_feed_redirect(url, final_url)
|
||||||
|
# Use final URL for validation
|
||||||
|
url = final_url
|
||||||
|
|
||||||
# Update last_successful_fetch since we got a 200 response
|
# Update last_successful_fetch since we got a 200 response
|
||||||
_update_feed_last_successful_fetch(url)
|
_update_feed_last_successful_fetch(url)
|
||||||
|
|
||||||
# Decode content as UTF-8 explicitly to avoid encoding issues
|
# Decode content as UTF-8 explicitly to avoid encoding issues
|
||||||
content = response.content.decode('utf-8')
|
content = response.content.decode("utf-8")
|
||||||
|
|
||||||
# Check if content has basic Org Social structure
|
# Check if content has basic Org Social structure
|
||||||
# At minimum should have at least one #+TITLE, #+NICK, or #+DESCRIPTION (case insensitive)
|
# At minimum should have at least one #+TITLE, #+NICK, or #+DESCRIPTION (case insensitive)
|
||||||
|
|
|
||||||
|
|
@ -15,181 +15,126 @@ def discover_feeds_from_relay_nodes():
|
||||||
Periodic task to discover new feeds from other Org Social Relay nodes.
|
Periodic task to discover new feeds from other Org Social Relay nodes.
|
||||||
|
|
||||||
This task:
|
This task:
|
||||||
1. Fetches the list of relay nodes from the public URL
|
1. Reads the list of relay nodes from a local file
|
||||||
2. Filters out our own domain to avoid self-discovery
|
2. Filters out our own domain to avoid self-discovery
|
||||||
3. Calls each relay node's /feeds endpoint to get their registered feeds
|
3. Calls each relay node's /feeds endpoint to get their registered feeds
|
||||||
4. Stores newly discovered feeds in our local database
|
4. Stores newly discovered feeds in our local database
|
||||||
"""
|
"""
|
||||||
import django
|
import django
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
django.setup()
|
django.setup()
|
||||||
|
|
||||||
from django.conf import settings
|
from django.conf import settings
|
||||||
|
from app.bridge.models import is_bridge_feed_url
|
||||||
from .models import Feed
|
from .models import Feed
|
||||||
from .parser import validate_org_social_feed
|
from .parser import validate_org_social_feed
|
||||||
|
|
||||||
# URLs to fetch feeds from
|
# Get the project root directory (where manage.py is located)
|
||||||
feed_sources = [
|
project_root = Path(__file__).resolve().parent.parent.parent
|
||||||
{
|
relay_list_path = project_root / "relay-list.txt"
|
||||||
"name": "relay nodes",
|
|
||||||
"url": "https://cdn.jsdelivr.net/gh/tanrax/org-social/org-social-relay-list.txt",
|
|
||||||
"type": "relay_nodes",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "public register",
|
|
||||||
"url": "https://raw.githubusercontent.com/tanrax/org-social/main/registers.txt",
|
|
||||||
"type": "direct_feeds",
|
|
||||||
},
|
|
||||||
]
|
|
||||||
|
|
||||||
total_discovered = 0
|
total_discovered = 0
|
||||||
|
|
||||||
for source in feed_sources:
|
logger.info(f"Reading relay nodes from: {relay_list_path}")
|
||||||
logger.info(f"Fetching feeds from {source['name']}: {source['url']}")
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Fetch the list
|
# Read the local file
|
||||||
response = requests.get(source["url"], timeout=5)
|
with open(relay_list_path, "r", encoding="utf-8") as f:
|
||||||
response.raise_for_status()
|
content = f.read()
|
||||||
|
|
||||||
# The file might be empty or contain one URL per line
|
# The file might be empty or contain one URL per line
|
||||||
urls = [line.strip() for line in response.text.split("\n") if line.strip()]
|
relay_nodes = [line.strip() for line in content.split("\n") if line.strip()]
|
||||||
|
|
||||||
if source["type"] == "direct_feeds":
|
# Filter out our own domain to avoid self-discovery
|
||||||
# For direct feeds (registers.txt), validate and add them directly
|
site_domain = settings.SITE_DOMAIN
|
||||||
logger.info(f"Found {len(urls)} direct feeds to validate")
|
filtered_nodes = []
|
||||||
|
for node_url in relay_nodes:
|
||||||
|
# Normalize the URL for comparison
|
||||||
|
normalized_node = (
|
||||||
|
node_url.replace("http://", "").replace("https://", "").strip("/")
|
||||||
|
)
|
||||||
|
normalized_site = site_domain.strip("/")
|
||||||
|
|
||||||
for feed_url in urls:
|
if normalized_node != normalized_site:
|
||||||
if not feed_url.strip():
|
filtered_nodes.append(node_url)
|
||||||
continue
|
else:
|
||||||
|
logger.info(f"Skipping own domain: {node_url}")
|
||||||
|
|
||||||
feed_url = feed_url.strip()
|
relay_nodes = filtered_nodes
|
||||||
|
|
||||||
# Check if we already have this feed
|
if not relay_nodes:
|
||||||
if Feed.objects.filter(url=feed_url).exists():
|
logger.info("No relay nodes found in the list after filtering own domain")
|
||||||
continue
|
return
|
||||||
|
|
||||||
# Validate the feed before adding it
|
logger.info(
|
||||||
logger.info(f"Validating direct feed: {feed_url}")
|
f"Found {len(relay_nodes)} relay nodes to check (excluding own domain)"
|
||||||
is_valid, error_message = validate_org_social_feed(feed_url)
|
)
|
||||||
|
|
||||||
if not is_valid:
|
for node_url in relay_nodes:
|
||||||
logger.warning(
|
try:
|
||||||
f"Skipping invalid direct feed {feed_url}: {error_message}"
|
# Ensure the URL has proper format
|
||||||
)
|
if not node_url.startswith(("http://", "https://")):
|
||||||
continue
|
node_url = f"http://{node_url}"
|
||||||
|
|
||||||
# Create the feed
|
# Call the /feeds endpoint on each relay node
|
||||||
try:
|
feeds_url = f"{node_url}/feeds"
|
||||||
Feed.objects.create(url=feed_url)
|
feeds_response = requests.get(feeds_url, timeout=10)
|
||||||
total_discovered += 1
|
feeds_response.raise_for_status()
|
||||||
logger.info(f"Added direct feed: {feed_url}")
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Failed to create direct feed {feed_url}: {e}")
|
|
||||||
|
|
||||||
elif source["type"] == "relay_nodes":
|
feeds_data = feeds_response.json()
|
||||||
# For relay nodes, get their feeds endpoints
|
|
||||||
relay_nodes = urls
|
|
||||||
|
|
||||||
# Filter out our own domain to avoid self-discovery
|
# Check if response has expected format
|
||||||
site_domain = settings.SITE_DOMAIN
|
if feeds_data.get("type") == "Success" and "data" in feeds_data:
|
||||||
filtered_nodes = []
|
feeds_list = feeds_data["data"]
|
||||||
for node_url in relay_nodes:
|
|
||||||
# Normalize the URL for comparison
|
|
||||||
normalized_node = (
|
|
||||||
node_url.replace("http://", "")
|
|
||||||
.replace("https://", "")
|
|
||||||
.strip("/")
|
|
||||||
)
|
|
||||||
normalized_site = site_domain.strip("/")
|
|
||||||
|
|
||||||
if normalized_node != normalized_site:
|
for feed_url in feeds_list:
|
||||||
filtered_nodes.append(node_url)
|
if isinstance(feed_url, str) and feed_url.strip():
|
||||||
else:
|
feed_url = feed_url.strip()
|
||||||
logger.info(f"Skipping own domain: {node_url}")
|
|
||||||
|
|
||||||
relay_nodes = filtered_nodes
|
# Bridge virtual feeds are not real accounts
|
||||||
|
if is_bridge_feed_url(feed_url):
|
||||||
|
continue
|
||||||
|
|
||||||
if not relay_nodes:
|
# Check if we already have this feed
|
||||||
logger.info(
|
if Feed.objects.filter(url=feed_url).exists():
|
||||||
"No relay nodes found in the list after filtering own domain"
|
continue
|
||||||
)
|
|
||||||
continue
|
|
||||||
|
|
||||||
logger.info(
|
# Validate the feed before adding it
|
||||||
f"Found {len(relay_nodes)} relay nodes to check (excluding own domain)"
|
logger.info(f"Validating discovered feed: {feed_url}")
|
||||||
)
|
is_valid, error_message = validate_org_social_feed(feed_url)
|
||||||
|
|
||||||
for node_url in relay_nodes:
|
if not is_valid:
|
||||||
try:
|
logger.warning(
|
||||||
# Ensure the URL has proper format
|
f"Skipping invalid feed {feed_url}: {error_message}"
|
||||||
if not node_url.startswith(("http://", "https://")):
|
)
|
||||||
node_url = f"http://{node_url}"
|
continue
|
||||||
|
|
||||||
# Call the /feeds endpoint on each relay node
|
# Create the feed
|
||||||
feeds_url = f"{node_url}/feeds"
|
try:
|
||||||
feeds_response = requests.get(feeds_url, timeout=10)
|
Feed.objects.create(url=feed_url)
|
||||||
feeds_response.raise_for_status()
|
total_discovered += 1
|
||||||
|
logger.info(
|
||||||
|
f"Discovered and validated new feed: {feed_url}"
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to create feed {feed_url}: {e}")
|
||||||
|
|
||||||
feeds_data = feeds_response.json()
|
logger.info(f"Successfully checked relay node: {node_url}")
|
||||||
|
|
||||||
# Check if response has expected format
|
except requests.RequestException as e:
|
||||||
if feeds_data.get("type") == "Success" and "data" in feeds_data:
|
logger.warning(f"Failed to fetch feeds from relay node {node_url}: {e}")
|
||||||
feeds_list = feeds_data["data"]
|
except ValueError as e:
|
||||||
|
logger.warning(f"Invalid JSON response from relay node {node_url}: {e}")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Unexpected error checking relay node {node_url}: {e}")
|
||||||
|
|
||||||
for feed_url in feeds_list:
|
except FileNotFoundError as e:
|
||||||
if isinstance(feed_url, str) and feed_url.strip():
|
logger.error(f"Relay list file not found: {relay_list_path}: {e}")
|
||||||
feed_url = feed_url.strip()
|
except IOError as e:
|
||||||
|
logger.error(f"Failed to read relay list from {relay_list_path}: {e}")
|
||||||
# Check if we already have this feed
|
except Exception as e:
|
||||||
if Feed.objects.filter(url=feed_url).exists():
|
logger.error(f"Unexpected error processing relay list: {e}")
|
||||||
continue
|
|
||||||
|
|
||||||
# Validate the feed before adding it
|
|
||||||
logger.info(
|
|
||||||
f"Validating discovered feed: {feed_url}"
|
|
||||||
)
|
|
||||||
is_valid, error_message = validate_org_social_feed(
|
|
||||||
feed_url
|
|
||||||
)
|
|
||||||
|
|
||||||
if not is_valid:
|
|
||||||
logger.warning(
|
|
||||||
f"Skipping invalid feed {feed_url}: {error_message}"
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Create the feed
|
|
||||||
try:
|
|
||||||
Feed.objects.create(url=feed_url)
|
|
||||||
total_discovered += 1
|
|
||||||
logger.info(
|
|
||||||
f"Discovered and validated new feed: {feed_url}"
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(
|
|
||||||
f"Failed to create feed {feed_url}: {e}"
|
|
||||||
)
|
|
||||||
|
|
||||||
logger.info(f"Successfully checked relay node: {node_url}")
|
|
||||||
|
|
||||||
except requests.RequestException as e:
|
|
||||||
logger.warning(
|
|
||||||
f"Failed to fetch feeds from relay node {node_url}: {e}"
|
|
||||||
)
|
|
||||||
except ValueError as e:
|
|
||||||
logger.warning(
|
|
||||||
f"Invalid JSON response from relay node {node_url}: {e}"
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(
|
|
||||||
f"Unexpected error checking relay node {node_url}: {e}"
|
|
||||||
)
|
|
||||||
|
|
||||||
except requests.RequestException as e:
|
|
||||||
logger.error(f"Failed to fetch {source['name']} from {source['url']}: {e}")
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Unexpected error processing {source['name']}: {e}")
|
|
||||||
|
|
||||||
logger.info(
|
logger.info(
|
||||||
f"Feed discovery completed. Total new feeds discovered: {total_discovered}"
|
f"Feed discovery completed. Total new feeds discovered: {total_discovered}"
|
||||||
|
|
@ -211,6 +156,7 @@ def discover_new_feeds_from_follows():
|
||||||
|
|
||||||
django.setup()
|
django.setup()
|
||||||
|
|
||||||
|
from app.bridge.models import is_bridge_feed_url
|
||||||
from .models import Feed, Profile, Follow
|
from .models import Feed, Profile, Follow
|
||||||
from .parser import parse_org_social, validate_org_social_feed
|
from .parser import parse_org_social, validate_org_social_feed
|
||||||
|
|
||||||
|
|
@ -232,10 +178,13 @@ def discover_new_feeds_from_follows():
|
||||||
|
|
||||||
for feed in all_feeds:
|
for feed in all_feeds:
|
||||||
try:
|
try:
|
||||||
# Parse the org social file
|
# Parse the org social file (may update feed URL if redirected)
|
||||||
parsed_data = parse_org_social(feed.url)
|
parsed_data = parse_org_social(feed.url)
|
||||||
successful_parses += 1
|
successful_parses += 1
|
||||||
|
|
||||||
|
# Refresh feed from database (URL may have changed due to redirect)
|
||||||
|
feed.refresh_from_db()
|
||||||
|
|
||||||
# Extract follow URLs from metadata
|
# Extract follow URLs from metadata
|
||||||
follows = parsed_data.get("metadata", {}).get("follows", [])
|
follows = parsed_data.get("metadata", {}).get("follows", [])
|
||||||
|
|
||||||
|
|
@ -258,6 +207,10 @@ def discover_new_feeds_from_follows():
|
||||||
"nick": metadata.get("nick", ""),
|
"nick": metadata.get("nick", ""),
|
||||||
"description": metadata.get("description", ""),
|
"description": metadata.get("description", ""),
|
||||||
"avatar": metadata.get("avatar", ""),
|
"avatar": metadata.get("avatar", ""),
|
||||||
|
"location": metadata.get("location", ""),
|
||||||
|
"birthday": metadata.get("birthday") or None,
|
||||||
|
"language": metadata.get("language", ""),
|
||||||
|
"pinned": metadata.get("pinned", ""),
|
||||||
"version": content_hash,
|
"version": content_hash,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
@ -276,7 +229,10 @@ def discover_new_feeds_from_follows():
|
||||||
# Check if feed already exists
|
# Check if feed already exists
|
||||||
existing_feed = Feed.objects.filter(url=follow_url).first()
|
existing_feed = Feed.objects.filter(url=follow_url).first()
|
||||||
|
|
||||||
if not existing_feed:
|
# Bridge virtual feeds help users connect to external
|
||||||
|
# content; the follow relationship is kept below, but the
|
||||||
|
# bridge is never registered as a real feed
|
||||||
|
if not existing_feed and not is_bridge_feed_url(follow_url):
|
||||||
# Validate the feed before adding it
|
# Validate the feed before adding it
|
||||||
logger.info(f"Validating discovered follow feed: {follow_url}")
|
logger.info(f"Validating discovered follow feed: {follow_url}")
|
||||||
is_valid, error_message = validate_org_social_feed(follow_url)
|
is_valid, error_message = validate_org_social_feed(follow_url)
|
||||||
|
|
@ -403,10 +359,13 @@ def scan_feeds():
|
||||||
|
|
||||||
for feed in all_feeds:
|
for feed in all_feeds:
|
||||||
try:
|
try:
|
||||||
# Parse the org social file
|
# Parse the org social file (may update feed URL if redirected)
|
||||||
parsed_data = parse_org_social(feed.url)
|
parsed_data = parse_org_social(feed.url)
|
||||||
successful_scans += 1
|
successful_scans += 1
|
||||||
|
|
||||||
|
# Refresh feed from database (URL may have changed due to redirect)
|
||||||
|
feed.refresh_from_db()
|
||||||
|
|
||||||
metadata = parsed_data.get("metadata", {})
|
metadata = parsed_data.get("metadata", {})
|
||||||
posts_data = parsed_data.get("posts", [])
|
posts_data = parsed_data.get("posts", [])
|
||||||
|
|
||||||
|
|
@ -422,6 +381,10 @@ def scan_feeds():
|
||||||
"nick": metadata.get("nick", ""),
|
"nick": metadata.get("nick", ""),
|
||||||
"description": metadata.get("description", ""),
|
"description": metadata.get("description", ""),
|
||||||
"avatar": metadata.get("avatar", ""),
|
"avatar": metadata.get("avatar", ""),
|
||||||
|
"location": metadata.get("location", ""),
|
||||||
|
"birthday": metadata.get("birthday") or None,
|
||||||
|
"language": metadata.get("language", ""),
|
||||||
|
"pinned": metadata.get("pinned", ""),
|
||||||
"version": content_hash,
|
"version": content_hash,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
@ -437,13 +400,14 @@ def scan_feeds():
|
||||||
profile.nick = metadata.get("nick", "")
|
profile.nick = metadata.get("nick", "")
|
||||||
profile.description = metadata.get("description", "")
|
profile.description = metadata.get("description", "")
|
||||||
profile.avatar = metadata.get("avatar", "")
|
profile.avatar = metadata.get("avatar", "")
|
||||||
|
profile.location = metadata.get("location", "")
|
||||||
|
profile.birthday = metadata.get("birthday") or None
|
||||||
|
profile.language = metadata.get("language", "")
|
||||||
|
profile.pinned = metadata.get("pinned", "")
|
||||||
profile.version = content_hash
|
profile.version = content_hash
|
||||||
profile.save()
|
profile.save()
|
||||||
profiles_updated += 1
|
profiles_updated += 1
|
||||||
logger.info(f"Updated profile: {profile.nick} ({feed.url})")
|
logger.info(f"Updated profile: {profile.nick} ({feed.url})")
|
||||||
else:
|
|
||||||
# No changes detected, skip processing
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Update profile relationships (clear and recreate)
|
# Update profile relationships (clear and recreate)
|
||||||
profile.links.all().delete()
|
profile.links.all().delete()
|
||||||
|
|
@ -476,6 +440,16 @@ def scan_feeds():
|
||||||
content = post_data.get("content", "")
|
content = post_data.get("content", "")
|
||||||
properties = post_data.get("properties", {})
|
properties = post_data.get("properties", {})
|
||||||
|
|
||||||
|
# Parse post_id as timestamp for created_at
|
||||||
|
# post_id is in RFC 3339 format (e.g., "2025-01-01T12:00:00+00:00")
|
||||||
|
post_created_at = timezone.now() # Default fallback
|
||||||
|
try:
|
||||||
|
post_created_at = date_parser.parse(post_id)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(
|
||||||
|
f"Failed to parse post_id {post_id} as timestamp: {e}. Using current time."
|
||||||
|
)
|
||||||
|
|
||||||
# Extract group name from GROUP property
|
# Extract group name from GROUP property
|
||||||
# Format: "Emacs https://org-social-relay.andros.dev" or just "Emacs"
|
# Format: "Emacs https://org-social-relay.andros.dev" or just "Emacs"
|
||||||
# Group names can have spaces and capitals - we slugify them
|
# Group names can have spaces and capitals - we slugify them
|
||||||
|
|
@ -510,14 +484,67 @@ def scan_feeds():
|
||||||
"reply_to": properties.get("reply_to", ""),
|
"reply_to": properties.get("reply_to", ""),
|
||||||
"mood": properties.get("mood", ""),
|
"mood": properties.get("mood", ""),
|
||||||
"group": group_slug,
|
"group": group_slug,
|
||||||
|
"include": properties.get("include", ""),
|
||||||
"poll_end": None,
|
"poll_end": None,
|
||||||
|
"created_at": post_created_at,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
if post_created:
|
if post_created:
|
||||||
posts_created += 1
|
posts_created += 1
|
||||||
|
|
||||||
|
# Queue outgoing webmentions for external links in the post
|
||||||
|
from .webmentions import queue_webmentions_for_post
|
||||||
|
|
||||||
|
try:
|
||||||
|
queue_webmentions_for_post(profile, post_id, content)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(
|
||||||
|
f"Failed to queue webmentions for post {post_id}: {e}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Publish notifications for NEW posts
|
||||||
|
from .notification_publisher import publish_notification
|
||||||
|
|
||||||
|
# Check if this is a reply (with or without mood/reaction)
|
||||||
|
if post.reply_to:
|
||||||
|
reply_to_parts = post.reply_to.split("#")
|
||||||
|
if len(reply_to_parts) == 2:
|
||||||
|
replied_feed_url = reply_to_parts[0]
|
||||||
|
|
||||||
|
# Determine if it's a reaction or a reply
|
||||||
|
if post.mood and post.mood.strip():
|
||||||
|
# It's a reaction
|
||||||
|
publish_notification(
|
||||||
|
target_feed_url=replied_feed_url,
|
||||||
|
notification_type="reaction",
|
||||||
|
post_url=f"{feed.url}#{post_id}",
|
||||||
|
emoji=post.mood,
|
||||||
|
parent=post.reply_to,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# It's a regular reply
|
||||||
|
publish_notification(
|
||||||
|
target_feed_url=replied_feed_url,
|
||||||
|
notification_type="reply",
|
||||||
|
post_url=f"{feed.url}#{post_id}",
|
||||||
|
parent=post.reply_to,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check if this is a boost
|
||||||
|
if post.include:
|
||||||
|
include_parts = post.include.split("#")
|
||||||
|
if len(include_parts) == 2:
|
||||||
|
boosted_feed_url = include_parts[0]
|
||||||
|
publish_notification(
|
||||||
|
target_feed_url=boosted_feed_url,
|
||||||
|
notification_type="boost",
|
||||||
|
post_url=f"{feed.url}#{post_id}",
|
||||||
|
boosted=post.include,
|
||||||
|
)
|
||||||
else:
|
else:
|
||||||
# Update existing post
|
# Update existing post
|
||||||
|
content_changed = post.content != content
|
||||||
post.content = content
|
post.content = content
|
||||||
post.language = properties.get("lang", "")
|
post.language = properties.get("lang", "")
|
||||||
post.tags = properties.get("tags", "")
|
post.tags = properties.get("tags", "")
|
||||||
|
|
@ -525,9 +552,22 @@ def scan_feeds():
|
||||||
post.reply_to = properties.get("reply_to", "")
|
post.reply_to = properties.get("reply_to", "")
|
||||||
post.mood = properties.get("mood", "")
|
post.mood = properties.get("mood", "")
|
||||||
post.group = group_slug
|
post.group = group_slug
|
||||||
|
post.include = properties.get("include", "")
|
||||||
post.save()
|
post.save()
|
||||||
posts_updated += 1
|
posts_updated += 1
|
||||||
|
|
||||||
|
# Links added by an edit get their webmention queued too;
|
||||||
|
# already-known (source, target) pairs are never re-sent
|
||||||
|
if content_changed:
|
||||||
|
from .webmentions import queue_webmentions_for_post
|
||||||
|
|
||||||
|
try:
|
||||||
|
queue_webmentions_for_post(profile, post_id, content)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(
|
||||||
|
f"Failed to queue webmentions for post {post_id}: {e}"
|
||||||
|
)
|
||||||
|
|
||||||
# Handle poll_end if present
|
# Handle poll_end if present
|
||||||
poll_end_str = properties.get("poll_end", "")
|
poll_end_str = properties.get("poll_end", "")
|
||||||
if poll_end_str:
|
if poll_end_str:
|
||||||
|
|
@ -584,13 +624,15 @@ def scan_feeds():
|
||||||
f"Failed to process poll vote for post {post_id}: {e}"
|
f"Failed to process poll vote for post {post_id}: {e}"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Handle mentions
|
# Handle mentions - FIXED to detect new mentions
|
||||||
mentions_data = post_data.get("mentions", [])
|
mentions_data = post_data.get("mentions", [])
|
||||||
if mentions_data:
|
if mentions_data:
|
||||||
# Clear existing mentions for this post
|
# Get existing mentions for this post
|
||||||
post.mentions.all().delete()
|
existing_mentions = set(
|
||||||
|
post.mentions.values_list("mentioned_profile__feed", flat=True)
|
||||||
|
)
|
||||||
|
|
||||||
# Create new mentions
|
# Process each mention
|
||||||
for mention_info in mentions_data:
|
for mention_info in mentions_data:
|
||||||
mention_url = mention_info.get("url", "").strip()
|
mention_url = mention_info.get("url", "").strip()
|
||||||
mention_nickname = mention_info.get("nickname", "").strip()
|
mention_nickname = mention_info.get("nickname", "").strip()
|
||||||
|
|
@ -607,16 +649,30 @@ def scan_feeds():
|
||||||
feed=base_mention_url
|
feed=base_mention_url
|
||||||
)
|
)
|
||||||
|
|
||||||
# Create the mention
|
# Only create if it doesn't exist (to detect new mentions)
|
||||||
Mention.objects.get_or_create(
|
if base_mention_url not in existing_mentions:
|
||||||
post=post,
|
mention, mention_created = (
|
||||||
mentioned_profile=mentioned_profile,
|
Mention.objects.get_or_create(
|
||||||
defaults={"nickname": mention_nickname},
|
post=post,
|
||||||
)
|
mentioned_profile=mentioned_profile,
|
||||||
|
defaults={"nickname": mention_nickname},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
# If this is a NEW mention, publish notification
|
||||||
|
if mention_created:
|
||||||
|
from .notification_publisher import (
|
||||||
|
publish_notification,
|
||||||
|
)
|
||||||
|
|
||||||
|
publish_notification(
|
||||||
|
target_feed_url=base_mention_url,
|
||||||
|
notification_type="mention",
|
||||||
|
post_url=f"{feed.url}#{post_id}",
|
||||||
|
)
|
||||||
|
|
||||||
except Profile.DoesNotExist:
|
except Profile.DoesNotExist:
|
||||||
# The mentioned profile doesn't exist in our database
|
# The mentioned profile doesn't exist in our database
|
||||||
# We could optionally log this or create a placeholder profile
|
|
||||||
logger.debug(f"Mentioned profile not found: {mention_url}")
|
logger.debug(f"Mentioned profile not found: {mention_url}")
|
||||||
continue
|
continue
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
|
@ -624,6 +680,27 @@ def scan_feeds():
|
||||||
f"Failed to create mention for {mention_url} in post {post_id}: {e}"
|
f"Failed to create mention for {mention_url} in post {post_id}: {e}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Detect and remove deleted posts
|
||||||
|
# Get all post IDs from the current feed scan
|
||||||
|
current_post_ids = {post_data.get("id", "") for post_data in posts_data}
|
||||||
|
current_post_ids.discard("") # Remove empty IDs
|
||||||
|
|
||||||
|
# Get all post IDs currently in database for this profile
|
||||||
|
existing_posts = Post.objects.filter(profile=profile)
|
||||||
|
existing_post_ids = set(existing_posts.values_list("post_id", flat=True))
|
||||||
|
|
||||||
|
# Find posts that are in DB but not in current feed (deleted posts)
|
||||||
|
deleted_post_ids = existing_post_ids - current_post_ids
|
||||||
|
|
||||||
|
if deleted_post_ids:
|
||||||
|
# Delete posts that no longer exist in the feed
|
||||||
|
deleted_count = Post.objects.filter(
|
||||||
|
profile=profile, post_id__in=deleted_post_ids
|
||||||
|
).delete()[0]
|
||||||
|
logger.info(
|
||||||
|
f"Removed {deleted_count} deleted post(s) from {feed.url}: {deleted_post_ids}"
|
||||||
|
)
|
||||||
|
|
||||||
except requests.RequestException as e:
|
except requests.RequestException as e:
|
||||||
failed_scans += 1
|
failed_scans += 1
|
||||||
logger.warning(f"Failed to fetch/parse feed {feed.url}: {e}")
|
logger.warning(f"Failed to fetch/parse feed {feed.url}: {e}")
|
||||||
|
|
@ -662,6 +739,128 @@ def scan_feeds():
|
||||||
logger.info("Cache cleared after feed scanning - next requests will get fresh data")
|
logger.info("Cache cleared after feed scanning - next requests will get fresh data")
|
||||||
|
|
||||||
|
|
||||||
|
WEBMENTION_MAX_ATTEMPTS = 5
|
||||||
|
WEBMENTION_BATCH_SIZE = 50
|
||||||
|
|
||||||
|
|
||||||
|
def _send_pending_webmentions_impl():
|
||||||
|
"""
|
||||||
|
Implementation of the webmention delivery logic, separated from the
|
||||||
|
periodic task to allow for easier testing.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
dict: Counters of sent / no_endpoint / failed webmentions
|
||||||
|
"""
|
||||||
|
from .models import OutgoingWebmention
|
||||||
|
from .webmentions import (
|
||||||
|
discover_webmention_endpoint,
|
||||||
|
is_safe_endpoint,
|
||||||
|
send_webmention,
|
||||||
|
)
|
||||||
|
|
||||||
|
now = timezone.now()
|
||||||
|
candidates = OutgoingWebmention.objects.filter(
|
||||||
|
status__in=[
|
||||||
|
OutgoingWebmention.STATUS_PENDING,
|
||||||
|
OutgoingWebmention.STATUS_FAILED,
|
||||||
|
],
|
||||||
|
attempts__lt=WEBMENTION_MAX_ATTEMPTS,
|
||||||
|
).order_by("created_at")[: WEBMENTION_BATCH_SIZE * 2]
|
||||||
|
|
||||||
|
counters = {"sent": 0, "no_endpoint": 0, "failed": 0}
|
||||||
|
processed = 0
|
||||||
|
|
||||||
|
for webmention in candidates:
|
||||||
|
if processed >= WEBMENTION_BATCH_SIZE:
|
||||||
|
break
|
||||||
|
|
||||||
|
# Exponential backoff between retries: 1h, 2h, 4h, 8h...
|
||||||
|
if webmention.last_attempt_at:
|
||||||
|
backoff = timedelta(hours=2 ** (webmention.attempts - 1))
|
||||||
|
if now < webmention.last_attempt_at + backoff:
|
||||||
|
continue
|
||||||
|
|
||||||
|
processed += 1
|
||||||
|
webmention.last_attempt_at = now
|
||||||
|
webmention.attempts += 1
|
||||||
|
|
||||||
|
try:
|
||||||
|
endpoint = webmention.endpoint or discover_webmention_endpoint(
|
||||||
|
webmention.target
|
||||||
|
)
|
||||||
|
|
||||||
|
if endpoint is None:
|
||||||
|
# Permanent: the target does not support webmentions
|
||||||
|
webmention.status = OutgoingWebmention.STATUS_NO_ENDPOINT
|
||||||
|
counters["no_endpoint"] += 1
|
||||||
|
webmention.save()
|
||||||
|
continue
|
||||||
|
|
||||||
|
if not is_safe_endpoint(endpoint):
|
||||||
|
logger.warning(
|
||||||
|
f"Rejecting unsafe webmention endpoint {endpoint} "
|
||||||
|
f"for target {webmention.target}"
|
||||||
|
)
|
||||||
|
webmention.status = OutgoingWebmention.STATUS_NO_ENDPOINT
|
||||||
|
counters["no_endpoint"] += 1
|
||||||
|
webmention.save()
|
||||||
|
continue
|
||||||
|
|
||||||
|
webmention.endpoint = endpoint
|
||||||
|
status_code = send_webmention(
|
||||||
|
endpoint, webmention.source, webmention.target
|
||||||
|
)
|
||||||
|
webmention.response_code = status_code
|
||||||
|
|
||||||
|
if 200 <= status_code < 300:
|
||||||
|
webmention.status = OutgoingWebmention.STATUS_SENT
|
||||||
|
counters["sent"] += 1
|
||||||
|
logger.info(
|
||||||
|
f"Webmention sent: {webmention.source} -> {webmention.target} "
|
||||||
|
f"({status_code})"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
webmention.status = OutgoingWebmention.STATUS_FAILED
|
||||||
|
counters["failed"] += 1
|
||||||
|
logger.warning(
|
||||||
|
f"Webmention rejected by {endpoint} with HTTP {status_code} "
|
||||||
|
f"({webmention.source} -> {webmention.target})"
|
||||||
|
)
|
||||||
|
|
||||||
|
except requests.RequestException as e:
|
||||||
|
webmention.status = OutgoingWebmention.STATUS_FAILED
|
||||||
|
counters["failed"] += 1
|
||||||
|
logger.warning(f"Webmention delivery error for {webmention.target}: {e}")
|
||||||
|
|
||||||
|
webmention.save()
|
||||||
|
|
||||||
|
if processed:
|
||||||
|
logger.info(
|
||||||
|
f"Webmention delivery completed. "
|
||||||
|
f"Sent: {counters['sent']}, "
|
||||||
|
f"No endpoint: {counters['no_endpoint']}, "
|
||||||
|
f"Failed: {counters['failed']}"
|
||||||
|
)
|
||||||
|
return counters
|
||||||
|
|
||||||
|
|
||||||
|
@periodic_task(crontab(minute="*/5")) # Run every 5 minutes
|
||||||
|
def send_pending_webmentions():
|
||||||
|
"""
|
||||||
|
Periodic task to deliver queued outgoing webmentions.
|
||||||
|
|
||||||
|
For each pending (or retriable failed) webmention this task discovers
|
||||||
|
the target's webmention endpoint and POSTs source/target to it, as
|
||||||
|
described in https://www.w3.org/TR/webmention/. Targets without an
|
||||||
|
endpoint are marked permanently so they are never fetched again.
|
||||||
|
"""
|
||||||
|
import django
|
||||||
|
|
||||||
|
django.setup()
|
||||||
|
|
||||||
|
return _send_pending_webmentions_impl()
|
||||||
|
|
||||||
|
|
||||||
def _cleanup_stale_feeds_impl():
|
def _cleanup_stale_feeds_impl():
|
||||||
"""
|
"""
|
||||||
Implementation of stale feed cleanup logic.
|
Implementation of stale feed cleanup logic.
|
||||||
|
|
|
||||||
231
app/feeds/test_discover_relay_nodes.py
Normal file
231
app/feeds/test_discover_relay_nodes.py
Normal file
|
|
@ -0,0 +1,231 @@
|
||||||
|
from django.test import TestCase
|
||||||
|
from django.conf import settings
|
||||||
|
from unittest.mock import patch, Mock
|
||||||
|
from app.feeds.models import Feed
|
||||||
|
from app.feeds.tasks import discover_feeds_from_relay_nodes
|
||||||
|
|
||||||
|
|
||||||
|
class DiscoverRelayNodesTest(TestCase):
|
||||||
|
"""Test cases for discover_feeds_from_relay_nodes task using Given/When/Then structure."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
"""Set up test fixtures."""
|
||||||
|
# Clear any existing feeds
|
||||||
|
Feed.objects.all().delete()
|
||||||
|
|
||||||
|
def test_discovers_feeds_from_real_relay_list(self):
|
||||||
|
"""Test that feeds are discovered from the actual relay list file."""
|
||||||
|
# Given: The real relay list file exists (data/relay-list.txt)
|
||||||
|
# And we mock the HTTP responses from relays
|
||||||
|
|
||||||
|
# Mock relay response
|
||||||
|
mock_relay_response = Mock()
|
||||||
|
mock_relay_response.status_code = 200
|
||||||
|
mock_relay_response.json.return_value = {
|
||||||
|
"type": "Success",
|
||||||
|
"data": [
|
||||||
|
"https://test-feed1.com/social.org",
|
||||||
|
"https://test-feed2.com/social.org",
|
||||||
|
],
|
||||||
|
}
|
||||||
|
mock_relay_response.raise_for_status = Mock()
|
||||||
|
|
||||||
|
# Mock feed validation responses
|
||||||
|
mock_feed_response = Mock()
|
||||||
|
mock_feed_response.status_code = 200
|
||||||
|
mock_feed_response.text = """#+TITLE: Test Feed
|
||||||
|
#+NICK: testuser
|
||||||
|
#+DESCRIPTION: Test description
|
||||||
|
|
||||||
|
* Posts
|
||||||
|
"""
|
||||||
|
mock_feed_response.content = mock_feed_response.text.encode("utf-8")
|
||||||
|
mock_feed_response.history = []
|
||||||
|
|
||||||
|
def mock_requests_get(url, timeout=None):
|
||||||
|
if "/feeds" in url:
|
||||||
|
return mock_relay_response
|
||||||
|
# Feed validation requests
|
||||||
|
mock_feed_response.url = url
|
||||||
|
mock_feed_response.raise_for_status = Mock()
|
||||||
|
return mock_feed_response
|
||||||
|
|
||||||
|
# When: We run the discovery task
|
||||||
|
with patch("requests.get", side_effect=mock_requests_get):
|
||||||
|
discover_feeds_from_relay_nodes()
|
||||||
|
|
||||||
|
# Then: Feeds should be discovered (the real relay list has relays)
|
||||||
|
# We can't predict exact count since it depends on real file content
|
||||||
|
# So we just verify the function runs without errors
|
||||||
|
self.assertGreaterEqual(Feed.objects.count(), 0)
|
||||||
|
|
||||||
|
def test_handles_relay_connection_failure(self):
|
||||||
|
"""Test that the task handles relay connection failures gracefully."""
|
||||||
|
# Given: The real relay list file exists
|
||||||
|
# And relay connections will fail
|
||||||
|
|
||||||
|
# Mock a connection error
|
||||||
|
with patch("requests.get", side_effect=Exception("Connection failed")):
|
||||||
|
# When: We run the discovery task
|
||||||
|
initial_count = Feed.objects.count()
|
||||||
|
discover_feeds_from_relay_nodes()
|
||||||
|
|
||||||
|
# Then: No new feeds should be created and no exception raised
|
||||||
|
self.assertEqual(Feed.objects.count(), initial_count)
|
||||||
|
|
||||||
|
def test_skips_existing_feeds(self):
|
||||||
|
"""Test that the task skips feeds that already exist."""
|
||||||
|
# Given: A feed already exists
|
||||||
|
Feed.objects.create(url="https://existing-feed.com/social.org")
|
||||||
|
initial_count = Feed.objects.count()
|
||||||
|
|
||||||
|
# Mock relay response with the existing feed
|
||||||
|
mock_relay_response = Mock()
|
||||||
|
mock_relay_response.status_code = 200
|
||||||
|
mock_relay_response.json.return_value = {
|
||||||
|
"type": "Success",
|
||||||
|
"data": [
|
||||||
|
"https://existing-feed.com/social.org", # Already exists
|
||||||
|
],
|
||||||
|
}
|
||||||
|
mock_relay_response.raise_for_status = Mock()
|
||||||
|
|
||||||
|
def mock_requests_get(url, timeout=None):
|
||||||
|
if "/feeds" in url:
|
||||||
|
return mock_relay_response
|
||||||
|
# Should not validate existing feed
|
||||||
|
if "existing-feed" in url:
|
||||||
|
self.fail("Should not validate existing feed")
|
||||||
|
return Mock()
|
||||||
|
|
||||||
|
# When: We run the discovery task
|
||||||
|
with patch("requests.get", side_effect=mock_requests_get):
|
||||||
|
discover_feeds_from_relay_nodes()
|
||||||
|
|
||||||
|
# Then: No new feeds should be added
|
||||||
|
self.assertEqual(Feed.objects.count(), initial_count)
|
||||||
|
|
||||||
|
def test_handles_invalid_feed_during_validation(self):
|
||||||
|
"""Test that the task skips invalid feeds during validation."""
|
||||||
|
# Given: The real relay list file exists
|
||||||
|
|
||||||
|
# Mock relay response with valid and invalid feeds
|
||||||
|
mock_relay_response = Mock()
|
||||||
|
mock_relay_response.status_code = 200
|
||||||
|
mock_relay_response.json.return_value = {
|
||||||
|
"type": "Success",
|
||||||
|
"data": [
|
||||||
|
"https://valid-feed.com/social.org",
|
||||||
|
"https://invalid-feed.com/social.org",
|
||||||
|
],
|
||||||
|
}
|
||||||
|
mock_relay_response.raise_for_status = Mock()
|
||||||
|
|
||||||
|
# Mock feed validation responses
|
||||||
|
def mock_requests_get(url, timeout=None):
|
||||||
|
if "/feeds" in url:
|
||||||
|
return mock_relay_response
|
||||||
|
|
||||||
|
# Valid feed response
|
||||||
|
if "valid-feed" in url:
|
||||||
|
mock_response = Mock()
|
||||||
|
mock_response.status_code = 200
|
||||||
|
mock_response.text = """#+TITLE: Valid Feed
|
||||||
|
#+NICK: validuser
|
||||||
|
#+DESCRIPTION: Valid description
|
||||||
|
|
||||||
|
* Posts
|
||||||
|
"""
|
||||||
|
mock_response.content = mock_response.text.encode("utf-8")
|
||||||
|
mock_response.url = url
|
||||||
|
mock_response.history = []
|
||||||
|
mock_response.raise_for_status = Mock()
|
||||||
|
return mock_response
|
||||||
|
|
||||||
|
# Invalid feed response (missing required fields)
|
||||||
|
if "invalid-feed" in url:
|
||||||
|
mock_response = Mock()
|
||||||
|
mock_response.status_code = 200
|
||||||
|
mock_response.text = "Invalid content without Org Social headers"
|
||||||
|
mock_response.content = mock_response.text.encode("utf-8")
|
||||||
|
mock_response.url = url
|
||||||
|
mock_response.history = []
|
||||||
|
mock_response.raise_for_status = Mock()
|
||||||
|
return mock_response
|
||||||
|
|
||||||
|
# Default response for any other URL
|
||||||
|
mock_response = Mock()
|
||||||
|
mock_response.status_code = 404
|
||||||
|
mock_response.raise_for_status = Mock(side_effect=Exception("404"))
|
||||||
|
return mock_response
|
||||||
|
|
||||||
|
# When: We run the discovery task
|
||||||
|
initial_count = Feed.objects.count()
|
||||||
|
with patch("requests.get", side_effect=mock_requests_get):
|
||||||
|
discover_feeds_from_relay_nodes()
|
||||||
|
|
||||||
|
# Then: Only the valid feed should be added (or none if validation failed for both)
|
||||||
|
# We verify that invalid feed was not added
|
||||||
|
self.assertFalse(
|
||||||
|
Feed.objects.filter(url="https://invalid-feed.com/social.org").exists()
|
||||||
|
)
|
||||||
|
# And if valid feed was processed, it should exist
|
||||||
|
# (we're more lenient here as the test DB might interfere)
|
||||||
|
if Feed.objects.count() > initial_count:
|
||||||
|
self.assertTrue(
|
||||||
|
Feed.objects.filter(url="https://valid-feed.com/social.org").exists()
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_filters_out_own_domain(self):
|
||||||
|
"""Test that the task filters out its own domain from relay list."""
|
||||||
|
# Given: The real relay list file exists
|
||||||
|
# We verify the function doesn't try to fetch from its own domain
|
||||||
|
|
||||||
|
own_domain = settings.SITE_DOMAIN
|
||||||
|
|
||||||
|
def mock_requests_get(url, timeout=None):
|
||||||
|
# Should not be called for our own domain
|
||||||
|
if own_domain in url and "/feeds" in url:
|
||||||
|
self.fail(f"Should not fetch from own domain: {url}")
|
||||||
|
|
||||||
|
# Mock response for other relays
|
||||||
|
mock_response = Mock()
|
||||||
|
mock_response.status_code = 200
|
||||||
|
mock_response.json.return_value = {"type": "Success", "data": []}
|
||||||
|
mock_response.raise_for_status = Mock()
|
||||||
|
return mock_response
|
||||||
|
|
||||||
|
# When: We run the discovery task
|
||||||
|
with patch("requests.get", side_effect=mock_requests_get):
|
||||||
|
discover_feeds_from_relay_nodes()
|
||||||
|
|
||||||
|
# Then: The function should complete without calling own domain
|
||||||
|
# (if it called own domain, the test would fail in mock_requests_get)
|
||||||
|
|
||||||
|
def test_skips_bridge_feeds_from_relay_nodes(self):
|
||||||
|
"""Test that bridge virtual feeds listed by other relays are skipped."""
|
||||||
|
# Given: A relay node listing only bridge virtual feeds
|
||||||
|
mock_relay_response = Mock()
|
||||||
|
mock_relay_response.status_code = 200
|
||||||
|
mock_relay_response.json.return_value = {
|
||||||
|
"type": "Success",
|
||||||
|
"data": [
|
||||||
|
"https://other-relay.org/bridge/rss/"
|
||||||
|
"?url=https%3A%2F%2Frss.arxiv.org%2Frss%2Fquant-ph",
|
||||||
|
"https://other-relay.org/bridge/activitypub/@user@instance.tld/",
|
||||||
|
],
|
||||||
|
}
|
||||||
|
mock_relay_response.raise_for_status = Mock()
|
||||||
|
|
||||||
|
def mock_requests_get(url, timeout=None):
|
||||||
|
if "/feeds" in url:
|
||||||
|
return mock_relay_response
|
||||||
|
# Bridge feeds must not even be validated
|
||||||
|
self.fail(f"Should not validate bridge feed: {url}")
|
||||||
|
|
||||||
|
# When: We run the discovery task
|
||||||
|
with patch("requests.get", side_effect=mock_requests_get):
|
||||||
|
discover_feeds_from_relay_nodes()
|
||||||
|
|
||||||
|
# Then: No bridge feed is registered
|
||||||
|
self.assertEqual(Feed.objects.count(), 0)
|
||||||
|
|
@ -25,6 +25,9 @@ class FeedCleanupTest(TestCase):
|
||||||
|
|
||||||
* Posts
|
* Posts
|
||||||
"""
|
"""
|
||||||
|
mock_response.content = mock_response.text.encode("utf-8")
|
||||||
|
mock_response.url = feed_url # No redirect
|
||||||
|
mock_response.history = [] # No redirect history
|
||||||
mock_response.raise_for_status = Mock()
|
mock_response.raise_for_status = Mock()
|
||||||
|
|
||||||
# When: We parse the feed and it returns HTTP 200
|
# When: We parse the feed and it returns HTTP 200
|
||||||
|
|
@ -55,6 +58,9 @@ class FeedCleanupTest(TestCase):
|
||||||
|
|
||||||
* Posts
|
* Posts
|
||||||
"""
|
"""
|
||||||
|
mock_response.content = mock_response.text.encode("utf-8")
|
||||||
|
mock_response.url = feed_url # No redirect
|
||||||
|
mock_response.history = [] # No redirect history
|
||||||
|
|
||||||
# When: We validate the feed and it returns HTTP 200
|
# When: We validate the feed and it returns HTTP 200
|
||||||
with patch("requests.get", return_value=mock_response):
|
with patch("requests.get", return_value=mock_response):
|
||||||
|
|
|
||||||
|
|
@ -277,7 +277,13 @@ class FeedsViewTest(TestCase):
|
||||||
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
|
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
|
||||||
self.assertEqual(response.data["type"], "Error")
|
self.assertEqual(response.data["type"], "Error")
|
||||||
self.assertIn("Invalid Org Social feed", response.data["errors"][0])
|
self.assertIn("Invalid Org Social feed", response.data["errors"][0])
|
||||||
self.assertIn("missing basic metadata", response.data["errors"][0])
|
# Accept either encoding errors or missing metadata errors
|
||||||
|
error_message = response.data["errors"][0]
|
||||||
|
self.assertTrue(
|
||||||
|
"missing basic metadata" in error_message
|
||||||
|
or "Validation error" in error_message,
|
||||||
|
f"Expected validation error, got: {error_message}",
|
||||||
|
)
|
||||||
self.assertIsNone(response.data["data"])
|
self.assertIsNone(response.data["data"])
|
||||||
|
|
||||||
def test_get_feeds_has_caching_headers(self):
|
def test_get_feeds_has_caching_headers(self):
|
||||||
|
|
@ -333,3 +339,43 @@ class FeedsViewTest(TestCase):
|
||||||
# Then: Should still have caching headers
|
# Then: Should still have caching headers
|
||||||
self.assertIn("ETag", response)
|
self.assertIn("ETag", response)
|
||||||
self.assertIn("Last-Modified", response)
|
self.assertIn("Last-Modified", response)
|
||||||
|
|
||||||
|
def test_get_feeds_excludes_bridge_feeds(self):
|
||||||
|
"""Test GET /feeds never lists bridge virtual feeds."""
|
||||||
|
# Given: A real feed and two bridge virtual feeds in the database
|
||||||
|
Feed.objects.create(url="https://example.com/social.org")
|
||||||
|
Feed.objects.create(
|
||||||
|
url="https://relay.org-social.org/bridge/rss/"
|
||||||
|
"?url=https%3A%2F%2Frss.arxiv.org%2Frss%2Fquant-ph"
|
||||||
|
)
|
||||||
|
Feed.objects.create(
|
||||||
|
url="https://relay.org-social.org/bridge/activitypub/@user@instance.tld/"
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: We request the feeds list
|
||||||
|
response = self.client.get(self.feeds_url)
|
||||||
|
|
||||||
|
# Then: Only the real feed is listed
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
self.assertEqual(response.data["data"], ["https://example.com/social.org"])
|
||||||
|
|
||||||
|
def test_post_bridge_feed_is_rejected(self):
|
||||||
|
"""Test POST /feeds rejects bridge virtual feeds."""
|
||||||
|
# Given: The URL of a bridge virtual feed
|
||||||
|
bridge_url = (
|
||||||
|
"https://relay.org-social.org/bridge/rss/"
|
||||||
|
"?url=https%3A%2F%2Fxkcd.com%2Frss.xml"
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: We try to register it
|
||||||
|
response = self.client.post(
|
||||||
|
self.feeds_url, {"feed": bridge_url}, format="json"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: The request is rejected and no feed is created
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn(
|
||||||
|
"Bridge virtual feeds cannot be registered", response.data["errors"]
|
||||||
|
)
|
||||||
|
self.assertFalse(Feed.objects.filter(url=bridge_url).exists())
|
||||||
|
|
|
||||||
|
|
@ -525,8 +525,8 @@ Great work!
|
||||||
# Then: Verify the bytes are correct UTF-8, not double-encoded
|
# Then: Verify the bytes are correct UTF-8, not double-encoded
|
||||||
# Correct UTF-8: f0 9f 99 8c f0 9f 8f bb (🙌🏻)
|
# Correct UTF-8: f0 9f 99 8c f0 9f 8f bb (🙌🏻)
|
||||||
# Double-encoded would be: c3 b0 c2 9f c2 99 c2 8c c3 b0 c2 9f c2 8f c2 bb
|
# Double-encoded would be: c3 b0 c2 9f c2 99 c2 8c c3 b0 c2 9f c2 8f c2 bb
|
||||||
mood_bytes = mood.encode('utf-8')
|
mood_bytes = mood.encode("utf-8")
|
||||||
self.assertEqual(mood_bytes.hex(), 'f09f998cf09f8fbb')
|
self.assertEqual(mood_bytes.hex(), "f09f998cf09f8fbb")
|
||||||
|
|
||||||
def test_parse_various_emojis_utf8(self):
|
def test_parse_various_emojis_utf8(self):
|
||||||
"""Test parsing various emojis to ensure UTF-8 encoding is preserved."""
|
"""Test parsing various emojis to ensure UTF-8 encoding is preserved."""
|
||||||
|
|
@ -588,9 +588,9 @@ Party post!
|
||||||
post = result["posts"][i]
|
post = result["posts"][i]
|
||||||
mood = post["properties"]["mood"]
|
mood = post["properties"]["mood"]
|
||||||
self.assertEqual(mood, expected_emoji)
|
self.assertEqual(mood, expected_emoji)
|
||||||
self.assertEqual(mood.encode('utf-8').hex(), expected_bytes)
|
self.assertEqual(mood.encode("utf-8").hex(), expected_bytes)
|
||||||
|
|
||||||
@patch('app.feeds.parser.requests.get')
|
@patch("app.feeds.parser.requests.get")
|
||||||
def test_parse_feed_with_missing_charset_header(self, mock_get):
|
def test_parse_feed_with_missing_charset_header(self, mock_get):
|
||||||
"""Test parsing a feed when server doesn't specify charset in Content-Type.
|
"""Test parsing a feed when server doesn't specify charset in Content-Type.
|
||||||
|
|
||||||
|
|
@ -615,15 +615,17 @@ Great work!
|
||||||
# Given: Mock response that simulates server without charset in Content-Type
|
# Given: Mock response that simulates server without charset in Content-Type
|
||||||
mock_response = Mock()
|
mock_response = Mock()
|
||||||
mock_response.status_code = 200
|
mock_response.status_code = 200
|
||||||
mock_response.encoding = 'ISO-8859-1' # requests default when no charset
|
mock_response.encoding = "ISO-8859-1" # requests default when no charset
|
||||||
mock_response.content = content_with_emoji.encode('utf-8') # Raw UTF-8 bytes
|
mock_response.content = content_with_emoji.encode("utf-8") # Raw UTF-8 bytes
|
||||||
|
mock_response.url = "https://example.com/social.org" # No redirect
|
||||||
|
mock_response.history = [] # No redirect history
|
||||||
mock_response.raise_for_status = Mock()
|
mock_response.raise_for_status = Mock()
|
||||||
mock_get.return_value = mock_response
|
mock_get.return_value = mock_response
|
||||||
|
|
||||||
# When: We parse the feed from URL
|
# When: We parse the feed from URL
|
||||||
from app.feeds.parser import parse_org_social
|
from app.feeds.parser import parse_org_social
|
||||||
|
|
||||||
result = parse_org_social('https://example.com/social.org')
|
result = parse_org_social("https://example.com/social.org")
|
||||||
|
|
||||||
# Then: The emoji should be correctly parsed (not double-encoded)
|
# Then: The emoji should be correctly parsed (not double-encoded)
|
||||||
self.assertEqual(len(result["posts"]), 1)
|
self.assertEqual(len(result["posts"]), 1)
|
||||||
|
|
@ -633,9 +635,336 @@ Great work!
|
||||||
self.assertEqual(mood, "🙌🏻")
|
self.assertEqual(mood, "🙌🏻")
|
||||||
|
|
||||||
# Then: Verify the bytes are correct UTF-8, not double-encoded
|
# Then: Verify the bytes are correct UTF-8, not double-encoded
|
||||||
mood_bytes = mood.encode('utf-8')
|
mood_bytes = mood.encode("utf-8")
|
||||||
self.assertEqual(mood_bytes.hex(), 'f09f998cf09f8fbb')
|
self.assertEqual(mood_bytes.hex(), "f09f998cf09f8fbb")
|
||||||
|
|
||||||
# Then: Verify we're using response.content, not response.text
|
# Then: Verify we're using response.content, not response.text
|
||||||
# This is critical to avoid double-encoding
|
# This is critical to avoid double-encoding
|
||||||
mock_get.assert_called_once_with('https://example.com/social.org', timeout=5)
|
from app.feeds.parser import FEED_FETCH_TIMEOUT
|
||||||
|
|
||||||
|
mock_get.assert_called_once_with(
|
||||||
|
"https://example.com/social.org", timeout=FEED_FETCH_TIMEOUT
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_parse_v16_metadata_fields(self):
|
||||||
|
"""Test parsing v1.6 metadata fields (LOCATION, BIRTHDAY, LANGUAGE, PINNED)."""
|
||||||
|
# Given: An org social content with v1.6 metadata fields
|
||||||
|
content = """#+TITLE: Test User Profile
|
||||||
|
#+NICK: test_user
|
||||||
|
#+DESCRIPTION: Test description
|
||||||
|
#+AVATAR: https://example.com/avatar.jpg
|
||||||
|
#+LOCATION: Valencia, Spain
|
||||||
|
#+BIRTHDAY: 1990-05-15
|
||||||
|
#+LANGUAGE: en es ca
|
||||||
|
#+PINNED: 2025-01-15T10:00:00+0100
|
||||||
|
|
||||||
|
* Posts
|
||||||
|
** 2025-01-15T10:00:00+0100
|
||||||
|
:PROPERTIES:
|
||||||
|
:END:
|
||||||
|
|
||||||
|
This is my pinned post.
|
||||||
|
|
||||||
|
** 2025-01-16T12:00:00+0100
|
||||||
|
:PROPERTIES:
|
||||||
|
:END:
|
||||||
|
|
||||||
|
This is a regular post.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# When: We parse the content
|
||||||
|
from app.feeds.parser import parse_org_social_content
|
||||||
|
|
||||||
|
result = parse_org_social_content(content)
|
||||||
|
|
||||||
|
# Then: All v1.6 metadata fields should be correctly parsed
|
||||||
|
self.assertEqual(result["metadata"]["location"], "Valencia, Spain")
|
||||||
|
self.assertEqual(result["metadata"]["birthday"], "1990-05-15")
|
||||||
|
self.assertEqual(result["metadata"]["language"], "en es ca")
|
||||||
|
self.assertEqual(result["metadata"]["pinned"], "2025-01-15T10:00:00+0100")
|
||||||
|
|
||||||
|
# Then: Posts should be parsed correctly
|
||||||
|
self.assertEqual(len(result["posts"]), 2)
|
||||||
|
|
||||||
|
def test_parse_invalid_birthday_is_dropped(self):
|
||||||
|
"""An invalid birthday format must be dropped, not abort the feed."""
|
||||||
|
# Given: A feed with a birthday that is not in YYYY-MM-DD format
|
||||||
|
content = """#+TITLE: Test User Profile
|
||||||
|
#+NICK: test_user
|
||||||
|
#+BIRTHDAY: 2003/06/17
|
||||||
|
|
||||||
|
* Posts
|
||||||
|
** 2025-01-15T10:00:00+0100
|
||||||
|
:PROPERTIES:
|
||||||
|
:END:
|
||||||
|
|
||||||
|
A post.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# When: We parse the content
|
||||||
|
from app.feeds.parser import parse_org_social_content
|
||||||
|
|
||||||
|
result = parse_org_social_content(content)
|
||||||
|
|
||||||
|
# Then: The malformed birthday is dropped but the feed is still parsed
|
||||||
|
self.assertEqual(result["metadata"]["birthday"], "")
|
||||||
|
self.assertEqual(result["metadata"]["nick"], "test_user")
|
||||||
|
self.assertEqual(len(result["posts"]), 1)
|
||||||
|
|
||||||
|
def test_parse_post_id_in_header(self):
|
||||||
|
"""Test parsing post ID from header (v1.6 feature)."""
|
||||||
|
# Given: An org social content with post ID in header
|
||||||
|
content = """#+TITLE: Test
|
||||||
|
#+NICK: test_user
|
||||||
|
|
||||||
|
* Posts
|
||||||
|
** 2025-01-15T10:00:00+0100
|
||||||
|
:PROPERTIES:
|
||||||
|
:LANG: en
|
||||||
|
:END:
|
||||||
|
|
||||||
|
This post has ID in the header.
|
||||||
|
|
||||||
|
**
|
||||||
|
:PROPERTIES:
|
||||||
|
:ID: 2025-01-16T12:00:00+0100
|
||||||
|
:END:
|
||||||
|
|
||||||
|
This post has ID in properties.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# When: We parse the content
|
||||||
|
from app.feeds.parser import parse_org_social_content
|
||||||
|
|
||||||
|
result = parse_org_social_content(content)
|
||||||
|
|
||||||
|
# Then: Both posts should have correct IDs
|
||||||
|
self.assertEqual(len(result["posts"]), 2)
|
||||||
|
|
||||||
|
# Then: First post should have ID from header
|
||||||
|
post1 = result["posts"][0]
|
||||||
|
self.assertEqual(post1["id"], "2025-01-15T10:00:00+0100")
|
||||||
|
|
||||||
|
# Then: Second post should have ID from properties
|
||||||
|
post2 = result["posts"][1]
|
||||||
|
self.assertEqual(post2["id"], "2025-01-16T12:00:00+0100")
|
||||||
|
|
||||||
|
def test_parse_post_id_priority_header_over_property(self):
|
||||||
|
"""Test that header ID takes priority over property ID (v1.6 spec)."""
|
||||||
|
# Given: An org social content with post ID in both header and properties
|
||||||
|
content = """#+TITLE: Test
|
||||||
|
#+NICK: test_user
|
||||||
|
|
||||||
|
* Posts
|
||||||
|
** 2025-01-15T10:00:00+0100
|
||||||
|
:PROPERTIES:
|
||||||
|
:ID: 2025-01-16T12:00:00+0100
|
||||||
|
:END:
|
||||||
|
|
||||||
|
This post has ID in both places. Header should take priority.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# When: We parse the content
|
||||||
|
from app.feeds.parser import parse_org_social_content
|
||||||
|
|
||||||
|
result = parse_org_social_content(content)
|
||||||
|
|
||||||
|
# Then: Post should have ID from header (priority)
|
||||||
|
self.assertEqual(len(result["posts"]), 1)
|
||||||
|
post = result["posts"][0]
|
||||||
|
self.assertEqual(post["id"], "2025-01-15T10:00:00+0100")
|
||||||
|
|
||||||
|
@patch("app.feeds.parser.requests.get")
|
||||||
|
def test_parse_feed_with_301_redirect(self, mock_get):
|
||||||
|
"""Test that feed redirects (301) are properly detected and handled."""
|
||||||
|
# Given: A feed URL that redirects to a new URL
|
||||||
|
old_url = "https://old.example.com/social.org"
|
||||||
|
new_url = "https://new.example.com/social.org"
|
||||||
|
|
||||||
|
content = """#+TITLE: Test User
|
||||||
|
#+NICK: test_user
|
||||||
|
|
||||||
|
* Posts
|
||||||
|
** 2025-01-15T10:00:00+0100
|
||||||
|
:PROPERTIES:
|
||||||
|
:END:
|
||||||
|
|
||||||
|
Test post
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Given: Mock response with redirect history
|
||||||
|
mock_redirect = Mock()
|
||||||
|
mock_redirect.status_code = 301
|
||||||
|
|
||||||
|
mock_response = Mock()
|
||||||
|
mock_response.status_code = 200
|
||||||
|
mock_response.content = content.encode("utf-8")
|
||||||
|
mock_response.url = new_url # Final URL after redirect
|
||||||
|
mock_response.history = [mock_redirect] # Redirect happened
|
||||||
|
mock_response.raise_for_status = Mock()
|
||||||
|
mock_get.return_value = mock_response
|
||||||
|
|
||||||
|
# When: We parse the feed from old URL
|
||||||
|
from app.feeds.parser import parse_org_social
|
||||||
|
|
||||||
|
result = parse_org_social(old_url)
|
||||||
|
|
||||||
|
# Then: The content should be parsed correctly
|
||||||
|
self.assertEqual(result["metadata"]["nick"], "test_user")
|
||||||
|
self.assertEqual(len(result["posts"]), 1)
|
||||||
|
|
||||||
|
@patch("app.feeds.parser.requests.get")
|
||||||
|
def test_validate_feed_with_301_redirect(self, mock_get):
|
||||||
|
"""Test that feed validation handles redirects properly."""
|
||||||
|
# Given: A feed URL that redirects to a new URL
|
||||||
|
old_url = "https://old.example.com/social.org"
|
||||||
|
new_url = "https://new.example.com/social.org"
|
||||||
|
|
||||||
|
content = """#+TITLE: Test User
|
||||||
|
#+NICK: test_user
|
||||||
|
|
||||||
|
* Posts
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Given: Mock response with redirect history
|
||||||
|
mock_redirect = Mock()
|
||||||
|
mock_redirect.status_code = 301
|
||||||
|
|
||||||
|
mock_response = Mock()
|
||||||
|
mock_response.status_code = 200
|
||||||
|
mock_response.content = content.encode("utf-8")
|
||||||
|
mock_response.url = new_url # Final URL after redirect
|
||||||
|
mock_response.history = [mock_redirect] # Redirect happened
|
||||||
|
mock_response.raise_for_status = Mock()
|
||||||
|
mock_get.return_value = mock_response
|
||||||
|
|
||||||
|
# When: We validate the feed
|
||||||
|
from app.feeds.parser import validate_org_social_feed
|
||||||
|
|
||||||
|
is_valid, error_message = validate_org_social_feed(old_url)
|
||||||
|
|
||||||
|
# Then: The feed should be valid
|
||||||
|
self.assertTrue(is_valid)
|
||||||
|
self.assertEqual(error_message, "")
|
||||||
|
|
||||||
|
|
||||||
|
class FeedRedirectHandlerTest(TestCase):
|
||||||
|
"""Test cases for feed redirect handling logic."""
|
||||||
|
|
||||||
|
def test_handle_redirect_updates_feed_url_when_only_old_exists(self):
|
||||||
|
"""Test updating feed URL when only old feed exists."""
|
||||||
|
# Given: A feed exists with old URL
|
||||||
|
from app.feeds.models import Feed, Profile
|
||||||
|
from app.feeds.parser import _handle_feed_redirect
|
||||||
|
|
||||||
|
old_url = "https://old.example.com/social.org"
|
||||||
|
new_url = "https://new.example.com/social.org"
|
||||||
|
|
||||||
|
Feed.objects.create(url=old_url)
|
||||||
|
Profile.objects.create(
|
||||||
|
feed=old_url, nick="testuser", title="Test User", version="v1"
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: Redirect is handled
|
||||||
|
_handle_feed_redirect(old_url, new_url)
|
||||||
|
|
||||||
|
# Then: Feed URL should be updated
|
||||||
|
self.assertFalse(Feed.objects.filter(url=old_url).exists())
|
||||||
|
self.assertTrue(Feed.objects.filter(url=new_url).exists())
|
||||||
|
|
||||||
|
# Then: Profile should point to new URL
|
||||||
|
profile = Profile.objects.get(feed=new_url)
|
||||||
|
self.assertEqual(profile.nick, "testuser")
|
||||||
|
|
||||||
|
def test_handle_redirect_merges_feeds_when_both_exist(self):
|
||||||
|
"""Test merging feeds when both old and new URLs exist."""
|
||||||
|
# Given: Both old and new feeds exist
|
||||||
|
from app.feeds.models import Feed, Profile, Post
|
||||||
|
from app.feeds.parser import _handle_feed_redirect
|
||||||
|
from django.utils import timezone
|
||||||
|
|
||||||
|
old_url = "https://old.example.com/social.org"
|
||||||
|
new_url = "https://new.example.com/social.org"
|
||||||
|
|
||||||
|
# Create old feed and profile
|
||||||
|
Feed.objects.create(url=old_url)
|
||||||
|
old_profile = Profile.objects.create(
|
||||||
|
feed=old_url, nick="olduser", title="Old User", version="v1"
|
||||||
|
)
|
||||||
|
Post.objects.create(
|
||||||
|
profile=old_profile,
|
||||||
|
post_id="2025-01-15T10:00:00+0100",
|
||||||
|
content="Old post",
|
||||||
|
created_at=timezone.now(),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create new feed and profile
|
||||||
|
Feed.objects.create(url=new_url)
|
||||||
|
new_profile = Profile.objects.create(
|
||||||
|
feed=new_url, nick="newuser", title="New User", version="v2"
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: Redirect is handled
|
||||||
|
_handle_feed_redirect(old_url, new_url)
|
||||||
|
|
||||||
|
# Then: Old feed should be deleted
|
||||||
|
self.assertFalse(Feed.objects.filter(url=old_url).exists())
|
||||||
|
self.assertTrue(Feed.objects.filter(url=new_url).exists())
|
||||||
|
|
||||||
|
# Then: Old profile should be deleted
|
||||||
|
self.assertFalse(Profile.objects.filter(feed=old_url).exists())
|
||||||
|
|
||||||
|
# Then: Post should be migrated to new profile
|
||||||
|
post = Post.objects.get(post_id="2025-01-15T10:00:00+0100")
|
||||||
|
self.assertEqual(post.profile, new_profile)
|
||||||
|
|
||||||
|
def test_handle_redirect_merges_mentions_without_duplicates(self):
|
||||||
|
"""Test that mentions are merged correctly, avoiding UNIQUE constraint errors."""
|
||||||
|
# Given: Both old and new profiles exist with mentions
|
||||||
|
from app.feeds.models import Feed, Profile, Post, Mention
|
||||||
|
from app.feeds.parser import _handle_feed_redirect
|
||||||
|
from django.utils import timezone
|
||||||
|
|
||||||
|
old_url = "https://old.example.com/social.org"
|
||||||
|
new_url = "https://new.example.com/social.org"
|
||||||
|
|
||||||
|
# Create old feed and profile
|
||||||
|
Feed.objects.create(url=old_url)
|
||||||
|
old_profile = Profile.objects.create(
|
||||||
|
feed=old_url, nick="olduser", title="Old User", version="v1"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create new feed and profile
|
||||||
|
Feed.objects.create(url=new_url)
|
||||||
|
new_profile = Profile.objects.create(
|
||||||
|
feed=new_url, nick="newuser", title="New User", version="v2"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create a third profile that mentions both old and new profiles
|
||||||
|
third_profile = Profile.objects.create(
|
||||||
|
feed="https://third.example.com/social.org",
|
||||||
|
nick="third",
|
||||||
|
title="Third User",
|
||||||
|
version="v1",
|
||||||
|
)
|
||||||
|
post = Post.objects.create(
|
||||||
|
profile=third_profile,
|
||||||
|
post_id="2025-01-15T10:00:00+0100",
|
||||||
|
content="Mentioning both profiles",
|
||||||
|
created_at=timezone.now(),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create mentions: one to old_profile, one to new_profile
|
||||||
|
Mention.objects.create(post=post, mentioned_profile=old_profile, nickname="old")
|
||||||
|
Mention.objects.create(post=post, mentioned_profile=new_profile, nickname="new")
|
||||||
|
|
||||||
|
# When: Redirect is handled (this should NOT raise UNIQUE constraint error)
|
||||||
|
_handle_feed_redirect(old_url, new_url)
|
||||||
|
|
||||||
|
# Then: Old profile should be deleted
|
||||||
|
self.assertFalse(Profile.objects.filter(feed=old_url).exists())
|
||||||
|
|
||||||
|
# Then: Only one mention should remain (the duplicate was deleted)
|
||||||
|
mentions = Mention.objects.filter(post=post)
|
||||||
|
self.assertEqual(mentions.count(), 1)
|
||||||
|
self.assertEqual(mentions.first().mentioned_profile, new_profile)
|
||||||
|
|
|
||||||
217
app/feeds/test_scan_feeds.py
Normal file
217
app/feeds/test_scan_feeds.py
Normal file
|
|
@ -0,0 +1,217 @@
|
||||||
|
from unittest.mock import Mock, patch
|
||||||
|
|
||||||
|
from django.test import TestCase
|
||||||
|
from app.feeds.models import Feed, Profile, Post
|
||||||
|
from app.feeds.tasks import scan_feeds
|
||||||
|
|
||||||
|
|
||||||
|
class PostDeletionDetectionTest(TestCase):
|
||||||
|
"""Test cases for detecting and removing deleted posts."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
"""Set up test fixtures."""
|
||||||
|
self.feed_url = "https://example.com/social.org"
|
||||||
|
self.feed = Feed.objects.create(url=self.feed_url)
|
||||||
|
self.profile = Profile.objects.create(
|
||||||
|
feed=self.feed_url,
|
||||||
|
nick="test_user",
|
||||||
|
title="Test User",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_detect_deleted_posts(self):
|
||||||
|
"""Test deletion detection logic."""
|
||||||
|
# Given: Profile has 3 posts in database
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile, post_id="2025-01-01T10:00:00+0100", content="Post 1"
|
||||||
|
)
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile, post_id="2025-01-02T10:00:00+0100", content="Post 2"
|
||||||
|
)
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile, post_id="2025-01-03T10:00:00+0100", content="Post 3"
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: Current feed only has 2 posts (simulating one was deleted)
|
||||||
|
current_post_ids = {"2025-01-01T10:00:00+0100", "2025-01-03T10:00:00+0100"}
|
||||||
|
|
||||||
|
# Get existing posts from database
|
||||||
|
existing_posts = Post.objects.filter(profile=self.profile)
|
||||||
|
existing_post_ids = set(existing_posts.values_list("post_id", flat=True))
|
||||||
|
|
||||||
|
# Find deleted posts
|
||||||
|
deleted_post_ids = existing_post_ids - current_post_ids
|
||||||
|
|
||||||
|
# Then: Should detect one deleted post
|
||||||
|
self.assertEqual(len(deleted_post_ids), 1)
|
||||||
|
self.assertIn("2025-01-02T10:00:00+0100", deleted_post_ids)
|
||||||
|
|
||||||
|
# When: Delete the posts that no longer exist
|
||||||
|
deleted_count = Post.objects.filter(
|
||||||
|
profile=self.profile, post_id__in=deleted_post_ids
|
||||||
|
).delete()[0]
|
||||||
|
|
||||||
|
# Then: One post should be deleted
|
||||||
|
self.assertEqual(deleted_count, 1)
|
||||||
|
|
||||||
|
# Then: Only 2 posts should remain
|
||||||
|
remaining_posts = Post.objects.filter(profile=self.profile)
|
||||||
|
self.assertEqual(remaining_posts.count(), 2)
|
||||||
|
|
||||||
|
# Then: Deleted post should not exist
|
||||||
|
self.assertFalse(
|
||||||
|
Post.objects.filter(
|
||||||
|
profile=self.profile, post_id="2025-01-02T10:00:00+0100"
|
||||||
|
).exists()
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: Other posts should still exist
|
||||||
|
self.assertTrue(
|
||||||
|
Post.objects.filter(
|
||||||
|
profile=self.profile, post_id="2025-01-01T10:00:00+0100"
|
||||||
|
).exists()
|
||||||
|
)
|
||||||
|
self.assertTrue(
|
||||||
|
Post.objects.filter(
|
||||||
|
profile=self.profile, post_id="2025-01-03T10:00:00+0100"
|
||||||
|
).exists()
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_no_posts_deleted_when_all_present(self):
|
||||||
|
"""Test that no posts are deleted when all posts are still in feed."""
|
||||||
|
# Given: Profile has 2 posts
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile, post_id="2025-01-01T10:00:00+0100", content="Post 1"
|
||||||
|
)
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile, post_id="2025-01-02T10:00:00+0100", content="Post 2"
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: Current feed still has both posts
|
||||||
|
current_post_ids = {"2025-01-01T10:00:00+0100", "2025-01-02T10:00:00+0100"}
|
||||||
|
|
||||||
|
# Get existing posts
|
||||||
|
existing_posts = Post.objects.filter(profile=self.profile)
|
||||||
|
existing_post_ids = set(existing_posts.values_list("post_id", flat=True))
|
||||||
|
|
||||||
|
# Find deleted posts
|
||||||
|
deleted_post_ids = existing_post_ids - current_post_ids
|
||||||
|
|
||||||
|
# Then: No posts should be detected as deleted
|
||||||
|
self.assertEqual(len(deleted_post_ids), 0)
|
||||||
|
|
||||||
|
# Then: Both posts should still exist
|
||||||
|
self.assertEqual(Post.objects.filter(profile=self.profile).count(), 2)
|
||||||
|
|
||||||
|
def test_all_posts_deleted(self):
|
||||||
|
"""Test handling when all posts are deleted from feed."""
|
||||||
|
# Given: Profile has 2 posts
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile, post_id="2025-01-01T10:00:00+0100", content="Post 1"
|
||||||
|
)
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile, post_id="2025-01-02T10:00:00+0100", content="Post 2"
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: Current feed has no posts
|
||||||
|
current_post_ids = set()
|
||||||
|
|
||||||
|
# Get existing posts
|
||||||
|
existing_posts = Post.objects.filter(profile=self.profile)
|
||||||
|
existing_post_ids = set(existing_posts.values_list("post_id", flat=True))
|
||||||
|
|
||||||
|
# Find deleted posts
|
||||||
|
deleted_post_ids = existing_post_ids - current_post_ids
|
||||||
|
|
||||||
|
# Then: All posts should be detected as deleted
|
||||||
|
self.assertEqual(len(deleted_post_ids), 2)
|
||||||
|
|
||||||
|
# When: Delete all posts
|
||||||
|
deleted_count = Post.objects.filter(
|
||||||
|
profile=self.profile, post_id__in=deleted_post_ids
|
||||||
|
).delete()[0]
|
||||||
|
|
||||||
|
# Then: 2 posts should be deleted
|
||||||
|
self.assertEqual(deleted_count, 2)
|
||||||
|
|
||||||
|
# Then: No posts should remain
|
||||||
|
self.assertEqual(Post.objects.filter(profile=self.profile).count(), 0)
|
||||||
|
|
||||||
|
def test_multiple_posts_deleted(self):
|
||||||
|
"""Test handling when multiple posts are deleted."""
|
||||||
|
# Given: Profile has 5 posts
|
||||||
|
for i in range(1, 6):
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile,
|
||||||
|
post_id=f"2025-01-0{i}T10:00:00+0100",
|
||||||
|
content=f"Post {i}",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: Current feed only has 2 posts (3 were deleted)
|
||||||
|
current_post_ids = {"2025-01-01T10:00:00+0100", "2025-01-05T10:00:00+0100"}
|
||||||
|
|
||||||
|
# Get existing posts
|
||||||
|
existing_posts = Post.objects.filter(profile=self.profile)
|
||||||
|
existing_post_ids = set(existing_posts.values_list("post_id", flat=True))
|
||||||
|
|
||||||
|
# Find deleted posts
|
||||||
|
deleted_post_ids = existing_post_ids - current_post_ids
|
||||||
|
|
||||||
|
# Then: 3 posts should be detected as deleted
|
||||||
|
self.assertEqual(len(deleted_post_ids), 3)
|
||||||
|
|
||||||
|
# When: Delete them
|
||||||
|
deleted_count = Post.objects.filter(
|
||||||
|
profile=self.profile, post_id__in=deleted_post_ids
|
||||||
|
).delete()[0]
|
||||||
|
|
||||||
|
# Then: 3 posts should be deleted
|
||||||
|
self.assertEqual(deleted_count, 3)
|
||||||
|
|
||||||
|
# Then: Only 2 posts should remain
|
||||||
|
self.assertEqual(Post.objects.filter(profile=self.profile).count(), 2)
|
||||||
|
|
||||||
|
|
||||||
|
class ScanFeedsRobustnessTest(TestCase):
|
||||||
|
"""End-to-end robustness tests for the scan_feeds task."""
|
||||||
|
|
||||||
|
@patch("app.feeds.parser.requests.get")
|
||||||
|
def test_invalid_birthday_does_not_abort_scan(self, mock_get):
|
||||||
|
"""A feed with a malformed birthday must still be scanned (regression).
|
||||||
|
|
||||||
|
Previously a value like "2003/06/17" reached the Profile.birthday
|
||||||
|
DateField and raised a ValidationError, aborting the whole feed scan on
|
||||||
|
every run. The birthday must now be dropped while posts are still saved.
|
||||||
|
"""
|
||||||
|
# Given: A feed whose birthday is not in YYYY-MM-DD format
|
||||||
|
feed_url = "https://host.example.org/ali/social.org"
|
||||||
|
Feed.objects.create(url=feed_url)
|
||||||
|
|
||||||
|
content = (
|
||||||
|
"#+TITLE: Ali\n"
|
||||||
|
"#+NICK: ali\n"
|
||||||
|
"#+BIRTHDAY: 2003/06/17\n"
|
||||||
|
"\n"
|
||||||
|
"* Posts\n"
|
||||||
|
"** 2025-01-01T10:00:00+0100\n"
|
||||||
|
":PROPERTIES:\n"
|
||||||
|
":END:\n"
|
||||||
|
"\n"
|
||||||
|
"Hello world\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
mock_response = Mock()
|
||||||
|
mock_response.status_code = 200
|
||||||
|
mock_response.content = content.encode("utf-8")
|
||||||
|
mock_response.url = feed_url # No redirect
|
||||||
|
mock_response.history = []
|
||||||
|
mock_response.raise_for_status = Mock()
|
||||||
|
mock_get.return_value = mock_response
|
||||||
|
|
||||||
|
# When: We scan all feeds
|
||||||
|
scan_feeds.call_local()
|
||||||
|
|
||||||
|
# Then: The profile is created with the bad birthday dropped, posts saved
|
||||||
|
profile = Profile.objects.get(feed=feed_url)
|
||||||
|
self.assertEqual(profile.nick, "ali")
|
||||||
|
self.assertIsNone(profile.birthday)
|
||||||
|
self.assertEqual(Post.objects.filter(profile=profile).count(), 1)
|
||||||
656
app/feeds/test_webmentions.py
Normal file
656
app/feeds/test_webmentions.py
Normal file
|
|
@ -0,0 +1,656 @@
|
||||||
|
from datetime import timedelta
|
||||||
|
from unittest.mock import Mock, patch
|
||||||
|
|
||||||
|
import requests
|
||||||
|
from django.test import TestCase
|
||||||
|
from django.utils import timezone
|
||||||
|
|
||||||
|
from app.feeds.models import Feed, OutgoingWebmention, Profile
|
||||||
|
from app.feeds.tasks import WEBMENTION_MAX_ATTEMPTS, _send_pending_webmentions_impl
|
||||||
|
from app.feeds.webmentions import (
|
||||||
|
discover_webmention_endpoint,
|
||||||
|
extract_external_urls,
|
||||||
|
is_safe_endpoint,
|
||||||
|
queue_webmentions_for_post,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ExtractExternalUrlsTest(TestCase):
|
||||||
|
"""Test cases for URL extraction from post content."""
|
||||||
|
|
||||||
|
def test_extract_org_link_with_description(self):
|
||||||
|
# Given: A post containing an Org link with description
|
||||||
|
content = "Great read: [[https://example.com/article][My article]]"
|
||||||
|
|
||||||
|
# When: URLs are extracted
|
||||||
|
result = extract_external_urls(content)
|
||||||
|
|
||||||
|
# Then: The linked URL is found without the description
|
||||||
|
self.assertEqual(result, ["https://example.com/article"])
|
||||||
|
|
||||||
|
def test_extract_org_link_without_description(self):
|
||||||
|
# Given: A post containing a plain Org link
|
||||||
|
content = "See [[https://example.com/article]] for details"
|
||||||
|
|
||||||
|
# When: URLs are extracted
|
||||||
|
result = extract_external_urls(content)
|
||||||
|
|
||||||
|
# Then: The linked URL is found
|
||||||
|
self.assertEqual(result, ["https://example.com/article"])
|
||||||
|
|
||||||
|
def test_extract_bare_url(self):
|
||||||
|
# Given: A post containing a bare URL
|
||||||
|
content = "Check https://example.com/post and tell me"
|
||||||
|
|
||||||
|
# When: URLs are extracted
|
||||||
|
result = extract_external_urls(content)
|
||||||
|
|
||||||
|
# Then: The bare URL is found
|
||||||
|
self.assertEqual(result, ["https://example.com/post"])
|
||||||
|
|
||||||
|
def test_bare_url_trailing_punctuation_stripped(self):
|
||||||
|
# Given: A bare URL wrapped in punctuation
|
||||||
|
content = "Nice one (https://example.com/post). Really!"
|
||||||
|
|
||||||
|
# When: URLs are extracted
|
||||||
|
result = extract_external_urls(content)
|
||||||
|
|
||||||
|
# Then: Trailing punctuation is not part of the URL
|
||||||
|
self.assertEqual(result, ["https://example.com/post"])
|
||||||
|
|
||||||
|
def test_url_with_query_string_and_fragment_is_preserved(self):
|
||||||
|
# Given: A URL with query string and fragment
|
||||||
|
content = "See https://example.com/post?id=1&lang=en#section-2"
|
||||||
|
|
||||||
|
# When: URLs are extracted
|
||||||
|
result = extract_external_urls(content)
|
||||||
|
|
||||||
|
# Then: Query string and fragment are kept intact
|
||||||
|
self.assertEqual(result, ["https://example.com/post?id=1&lang=en#section-2"])
|
||||||
|
|
||||||
|
def test_org_social_mentions_are_ignored(self):
|
||||||
|
# Given: A post with an Org Social mention and a normal link
|
||||||
|
content = (
|
||||||
|
"Hi [[org-social:https://other.example/social.org][bob]] look at "
|
||||||
|
"[[https://example.com/article][this]]"
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: URLs are extracted
|
||||||
|
result = extract_external_urls(content)
|
||||||
|
|
||||||
|
# Then: Only the normal link is found, the mention is skipped
|
||||||
|
self.assertEqual(result, ["https://example.com/article"])
|
||||||
|
|
||||||
|
def test_duplicate_urls_are_deduplicated(self):
|
||||||
|
# Given: A post repeating the same URL as Org link and bare URL
|
||||||
|
content = (
|
||||||
|
"https://example.com/a and again [[https://example.com/a][same]] "
|
||||||
|
"plus https://example.com/b"
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: URLs are extracted
|
||||||
|
result = extract_external_urls(content)
|
||||||
|
|
||||||
|
# Then: Each URL appears once, in order of appearance
|
||||||
|
self.assertEqual(result, ["https://example.com/a", "https://example.com/b"])
|
||||||
|
|
||||||
|
def test_empty_or_missing_content_returns_no_urls(self):
|
||||||
|
# Given: Empty and missing content
|
||||||
|
# When: URLs are extracted
|
||||||
|
# Then: No URLs are found and nothing breaks
|
||||||
|
self.assertEqual(extract_external_urls(""), [])
|
||||||
|
self.assertEqual(extract_external_urls(None), [])
|
||||||
|
|
||||||
|
def test_non_http_schemes_are_ignored(self):
|
||||||
|
# Given: A post with non-http(s) URIs only
|
||||||
|
content = "Write to mailto:me@example.com or ftp://example.com/file"
|
||||||
|
|
||||||
|
# When: URLs are extracted
|
||||||
|
result = extract_external_urls(content)
|
||||||
|
|
||||||
|
# Then: Nothing is extracted
|
||||||
|
self.assertEqual(result, [])
|
||||||
|
|
||||||
|
|
||||||
|
def _mock_response(
|
||||||
|
url="https://example.com/article",
|
||||||
|
headers=None,
|
||||||
|
body=b"",
|
||||||
|
encoding="utf-8",
|
||||||
|
):
|
||||||
|
"""Build a minimal mock of a streamed requests.Response."""
|
||||||
|
response = Mock()
|
||||||
|
response.url = url
|
||||||
|
response.headers = headers or {}
|
||||||
|
response.encoding = encoding
|
||||||
|
response.raw = Mock()
|
||||||
|
response.raw.read = Mock(return_value=body)
|
||||||
|
return response
|
||||||
|
|
||||||
|
|
||||||
|
class DiscoverWebmentionEndpointTest(TestCase):
|
||||||
|
"""Test cases for endpoint discovery per the W3C spec."""
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.requests.get")
|
||||||
|
def test_link_header_discovery(self, mock_get):
|
||||||
|
# Given: A target advertising its endpoint in the HTTP Link header
|
||||||
|
mock_get.return_value = _mock_response(
|
||||||
|
headers={
|
||||||
|
"Link": '<https://example.com/wm>; rel="webmention"',
|
||||||
|
"Content-Type": "text/html",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The endpoint is discovered
|
||||||
|
endpoint = discover_webmention_endpoint("https://example.com/article")
|
||||||
|
|
||||||
|
# Then: The Link header endpoint is returned
|
||||||
|
self.assertEqual(endpoint, "https://example.com/wm")
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.requests.get")
|
||||||
|
def test_link_header_takes_precedence_over_html(self, mock_get):
|
||||||
|
# Given: A target with endpoints both in the Link header and the HTML
|
||||||
|
mock_get.return_value = _mock_response(
|
||||||
|
headers={
|
||||||
|
"Link": '<https://example.com/header-wm>; rel="webmention"',
|
||||||
|
"Content-Type": "text/html",
|
||||||
|
},
|
||||||
|
body=b'<link rel="webmention" href="https://example.com/html-wm">',
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The endpoint is discovered
|
||||||
|
endpoint = discover_webmention_endpoint("https://example.com/article")
|
||||||
|
|
||||||
|
# Then: The Link header wins, as the spec requires
|
||||||
|
self.assertEqual(endpoint, "https://example.com/header-wm")
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.requests.get")
|
||||||
|
def test_link_header_with_multiple_rels(self, mock_get):
|
||||||
|
# Given: A Link header whose rel lists several values
|
||||||
|
mock_get.return_value = _mock_response(
|
||||||
|
headers={
|
||||||
|
"Link": '<https://example.com/wm>; rel="webmention somethingelse"',
|
||||||
|
"Content-Type": "text/html",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The endpoint is discovered
|
||||||
|
endpoint = discover_webmention_endpoint("https://example.com/article")
|
||||||
|
|
||||||
|
# Then: The webmention rel is still recognized
|
||||||
|
self.assertEqual(endpoint, "https://example.com/wm")
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.requests.get")
|
||||||
|
def test_html_link_element_discovery(self, mock_get):
|
||||||
|
# Given: A target advertising a relative endpoint in a <link> element
|
||||||
|
mock_get.return_value = _mock_response(
|
||||||
|
headers={"Content-Type": "text/html; charset=utf-8"},
|
||||||
|
body=b'<html><head><link rel="webmention" href="/wm"></head></html>',
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The endpoint is discovered
|
||||||
|
endpoint = discover_webmention_endpoint("https://example.com/article")
|
||||||
|
|
||||||
|
# Then: The relative href is resolved against the target
|
||||||
|
self.assertEqual(endpoint, "https://example.com/wm")
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.requests.get")
|
||||||
|
def test_html_anchor_element_discovery(self, mock_get):
|
||||||
|
# Given: A target advertising its endpoint in an <a> element
|
||||||
|
mock_get.return_value = _mock_response(
|
||||||
|
headers={"Content-Type": "text/html"},
|
||||||
|
body=b'<body><a rel="webmention" href="https://wm.example.org/ep">wm</a></body>',
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The endpoint is discovered
|
||||||
|
endpoint = discover_webmention_endpoint("https://example.com/article")
|
||||||
|
|
||||||
|
# Then: The anchor endpoint is returned
|
||||||
|
self.assertEqual(endpoint, "https://wm.example.org/ep")
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.requests.get")
|
||||||
|
def test_first_element_in_document_order_wins(self, mock_get):
|
||||||
|
# Given: Both an <a> and a <link> endpoint, the <a> appearing first
|
||||||
|
mock_get.return_value = _mock_response(
|
||||||
|
headers={"Content-Type": "text/html"},
|
||||||
|
body=(
|
||||||
|
b'<a rel="webmention" href="/first">a</a>'
|
||||||
|
b'<link rel="webmention" href="/second">'
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The endpoint is discovered
|
||||||
|
endpoint = discover_webmention_endpoint("https://example.com/article")
|
||||||
|
|
||||||
|
# Then: The first element in document order wins
|
||||||
|
self.assertEqual(endpoint, "https://example.com/first")
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.requests.get")
|
||||||
|
def test_empty_href_resolves_to_page_url(self, mock_get):
|
||||||
|
# Given: An endpoint advertised with an empty href
|
||||||
|
mock_get.return_value = _mock_response(
|
||||||
|
url="https://example.com/article",
|
||||||
|
headers={"Content-Type": "text/html"},
|
||||||
|
body=b'<link rel="webmention" href="">',
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The endpoint is discovered
|
||||||
|
endpoint = discover_webmention_endpoint("https://example.com/article")
|
||||||
|
|
||||||
|
# Then: It resolves to the page URL itself
|
||||||
|
self.assertEqual(endpoint, "https://example.com/article")
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.requests.get")
|
||||||
|
def test_relative_endpoint_resolved_after_redirect(self, mock_get):
|
||||||
|
# Given: A target that redirected and advertises a relative endpoint
|
||||||
|
mock_get.return_value = _mock_response(
|
||||||
|
url="https://final.example.com/page",
|
||||||
|
headers={"Content-Type": "text/html"},
|
||||||
|
body=b'<link rel="webmention" href="wm-endpoint">',
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The endpoint is discovered
|
||||||
|
endpoint = discover_webmention_endpoint("https://example.com/article")
|
||||||
|
|
||||||
|
# Then: The endpoint resolves against the final URL after redirects
|
||||||
|
self.assertEqual(endpoint, "https://final.example.com/wm-endpoint")
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.requests.get")
|
||||||
|
def test_no_endpoint_returns_none(self, mock_get):
|
||||||
|
# Given: A target without any webmention endpoint
|
||||||
|
mock_get.return_value = _mock_response(
|
||||||
|
headers={"Content-Type": "text/html"},
|
||||||
|
body=b"<html><body>No webmention here</body></html>",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The endpoint is discovered
|
||||||
|
endpoint = discover_webmention_endpoint("https://example.com/article")
|
||||||
|
|
||||||
|
# Then: No endpoint is found
|
||||||
|
self.assertIsNone(endpoint)
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.requests.get")
|
||||||
|
def test_non_html_content_without_link_header_returns_none(self, mock_get):
|
||||||
|
# Given: A non-HTML target without a Link header
|
||||||
|
mock_get.return_value = _mock_response(
|
||||||
|
headers={"Content-Type": "application/pdf"},
|
||||||
|
body=b"%PDF-1.4",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The endpoint is discovered
|
||||||
|
endpoint = discover_webmention_endpoint("https://example.com/doc.pdf")
|
||||||
|
|
||||||
|
# Then: No endpoint is found and the body is never parsed as HTML
|
||||||
|
self.assertIsNone(endpoint)
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.requests.get")
|
||||||
|
def test_rel_without_webmention_is_ignored(self, mock_get):
|
||||||
|
# Given: A page whose only <link> has an unrelated rel
|
||||||
|
mock_get.return_value = _mock_response(
|
||||||
|
headers={"Content-Type": "text/html"},
|
||||||
|
body=b'<link rel="stylesheet" href="/style.css">',
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The endpoint is discovered
|
||||||
|
endpoint = discover_webmention_endpoint("https://example.com/article")
|
||||||
|
|
||||||
|
# Then: No endpoint is found
|
||||||
|
self.assertIsNone(endpoint)
|
||||||
|
|
||||||
|
|
||||||
|
class IsSafeEndpointTest(TestCase):
|
||||||
|
"""Test cases for endpoint safety checks (spec section 4.3)."""
|
||||||
|
|
||||||
|
def test_rejects_non_http_scheme(self):
|
||||||
|
# Given: Endpoints with non-http(s) schemes
|
||||||
|
# When: They are checked
|
||||||
|
# Then: They are rejected
|
||||||
|
self.assertFalse(is_safe_endpoint("ftp://example.com/wm"))
|
||||||
|
self.assertFalse(is_safe_endpoint("file:///etc/passwd"))
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.socket.getaddrinfo")
|
||||||
|
def test_rejects_loopback(self, mock_getaddrinfo):
|
||||||
|
# Given: An endpoint resolving to a loopback address
|
||||||
|
mock_getaddrinfo.return_value = [(2, 1, 6, "", ("127.0.0.1", 80))]
|
||||||
|
|
||||||
|
# When/Then: It is rejected
|
||||||
|
self.assertFalse(is_safe_endpoint("http://localhost/wm"))
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.socket.getaddrinfo")
|
||||||
|
def test_rejects_private_address(self, mock_getaddrinfo):
|
||||||
|
# Given: An endpoint resolving to a private address
|
||||||
|
mock_getaddrinfo.return_value = [(2, 1, 6, "", ("192.168.1.10", 80))]
|
||||||
|
|
||||||
|
# When/Then: It is rejected
|
||||||
|
self.assertFalse(is_safe_endpoint("https://internal.example.com/wm"))
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.socket.getaddrinfo")
|
||||||
|
def test_accepts_public_address(self, mock_getaddrinfo):
|
||||||
|
# Given: An endpoint resolving to a public address
|
||||||
|
mock_getaddrinfo.return_value = [(2, 1, 6, "", ("93.184.216.34", 443))]
|
||||||
|
|
||||||
|
# When/Then: It is accepted
|
||||||
|
self.assertTrue(is_safe_endpoint("https://example.com/wm"))
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.socket.getaddrinfo")
|
||||||
|
def test_rejects_unresolvable_host(self, mock_getaddrinfo):
|
||||||
|
# Given: An endpoint whose host does not resolve
|
||||||
|
import socket as socket_module
|
||||||
|
|
||||||
|
mock_getaddrinfo.side_effect = socket_module.gaierror()
|
||||||
|
|
||||||
|
# When/Then: It is rejected
|
||||||
|
self.assertFalse(is_safe_endpoint("https://nope.invalid/wm"))
|
||||||
|
|
||||||
|
|
||||||
|
class QueueWebmentionsForPostTest(TestCase):
|
||||||
|
"""Test cases for queueing outgoing webmentions during feed scans."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
"""Set up test fixtures."""
|
||||||
|
self.feed_url = "https://example.com/social.org"
|
||||||
|
self.feed = Feed.objects.create(url=self.feed_url)
|
||||||
|
self.profile = Profile.objects.create(
|
||||||
|
feed=self.feed_url, nick="alice", title="Alice"
|
||||||
|
)
|
||||||
|
self.post_id = "2026-01-01T10:00:00+00:00"
|
||||||
|
|
||||||
|
def test_queues_external_urls_as_pending(self):
|
||||||
|
# Given: A post linking to an external article
|
||||||
|
content = "I wrote about it: [[https://blog.example.org/post][my post]]"
|
||||||
|
|
||||||
|
# When: Webmentions are queued for the post
|
||||||
|
queued = queue_webmentions_for_post(self.profile, self.post_id, content)
|
||||||
|
|
||||||
|
# Then: One pending webmention exists with the post as source
|
||||||
|
self.assertEqual(queued, 1)
|
||||||
|
webmention = OutgoingWebmention.objects.get()
|
||||||
|
self.assertEqual(webmention.source, f"{self.feed_url}#{self.post_id}")
|
||||||
|
self.assertEqual(webmention.target, "https://blog.example.org/post")
|
||||||
|
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_PENDING)
|
||||||
|
|
||||||
|
def test_rescanning_same_post_does_not_duplicate(self):
|
||||||
|
# Given: A post whose webmention was already queued
|
||||||
|
content = "Look at https://blog.example.org/post please"
|
||||||
|
queue_webmentions_for_post(self.profile, self.post_id, content)
|
||||||
|
|
||||||
|
# When: The same post is queued again (rescan)
|
||||||
|
queued = queue_webmentions_for_post(self.profile, self.post_id, content)
|
||||||
|
|
||||||
|
# Then: Nothing new is queued
|
||||||
|
self.assertEqual(queued, 0)
|
||||||
|
self.assertEqual(OutgoingWebmention.objects.count(), 1)
|
||||||
|
|
||||||
|
def test_rescan_does_not_reset_sent_status(self):
|
||||||
|
# Given: A webmention already delivered for the post
|
||||||
|
content = "Look at https://blog.example.org/post please"
|
||||||
|
queue_webmentions_for_post(self.profile, self.post_id, content)
|
||||||
|
OutgoingWebmention.objects.update(status=OutgoingWebmention.STATUS_SENT)
|
||||||
|
|
||||||
|
# When: The same post is queued again (rescan)
|
||||||
|
queue_webmentions_for_post(self.profile, self.post_id, content)
|
||||||
|
|
||||||
|
# Then: The delivered webmention keeps its sent status
|
||||||
|
self.assertEqual(
|
||||||
|
OutgoingWebmention.objects.get().status, OutgoingWebmention.STATUS_SENT
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_edited_post_queues_only_new_urls(self):
|
||||||
|
# Given: A post whose original link was already delivered
|
||||||
|
queue_webmentions_for_post(
|
||||||
|
self.profile, self.post_id, "First link https://blog.example.org/a"
|
||||||
|
)
|
||||||
|
OutgoingWebmention.objects.update(status=OutgoingWebmention.STATUS_SENT)
|
||||||
|
|
||||||
|
# When: The post is edited adding a second link
|
||||||
|
queued = queue_webmentions_for_post(
|
||||||
|
self.profile,
|
||||||
|
self.post_id,
|
||||||
|
"First link https://blog.example.org/a and https://blog.example.org/b",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: Only the added link is queued, the delivered one is untouched
|
||||||
|
self.assertEqual(queued, 1)
|
||||||
|
self.assertEqual(OutgoingWebmention.objects.count(), 2)
|
||||||
|
self.assertEqual(
|
||||||
|
OutgoingWebmention.objects.get(target="https://blog.example.org/a").status,
|
||||||
|
OutgoingWebmention.STATUS_SENT,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_registered_feeds_are_excluded(self):
|
||||||
|
# Given: A post linking to a registered Org Social feed and to an article
|
||||||
|
Feed.objects.create(url="https://friend.example.org/social.org")
|
||||||
|
content = (
|
||||||
|
"Read https://friend.example.org/social.org#2026-01-01T00:00:00+00:00 "
|
||||||
|
"and https://blog.example.org/post"
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: Webmentions are queued for the post
|
||||||
|
queued = queue_webmentions_for_post(self.profile, self.post_id, content)
|
||||||
|
|
||||||
|
# Then: Only the article is queued, the feed link is handled natively
|
||||||
|
self.assertEqual(queued, 1)
|
||||||
|
self.assertEqual(
|
||||||
|
OutgoingWebmention.objects.get().target, "https://blog.example.org/post"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_own_feed_is_excluded(self):
|
||||||
|
# Given: A post linking to the author's own feed
|
||||||
|
content = f"Self reference {self.feed_url}#2025-12-31T00:00:00+00:00"
|
||||||
|
|
||||||
|
# When: Webmentions are queued for the post
|
||||||
|
queued = queue_webmentions_for_post(self.profile, self.post_id, content)
|
||||||
|
|
||||||
|
# Then: Nothing is queued
|
||||||
|
self.assertEqual(queued, 0)
|
||||||
|
|
||||||
|
def test_post_without_urls_queues_nothing(self):
|
||||||
|
# Given: A post with no URLs at all
|
||||||
|
content = "Just text"
|
||||||
|
|
||||||
|
# When: Webmentions are queued for the post
|
||||||
|
queued = queue_webmentions_for_post(self.profile, self.post_id, content)
|
||||||
|
|
||||||
|
# Then: Nothing is queued
|
||||||
|
self.assertEqual(queued, 0)
|
||||||
|
self.assertEqual(OutgoingWebmention.objects.count(), 0)
|
||||||
|
|
||||||
|
|
||||||
|
class ScanFeedsWebmentionIntegrationTest(TestCase):
|
||||||
|
"""Integration tests: scan_feeds queues webmentions exactly once."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
"""Set up test fixtures."""
|
||||||
|
self.feed_url = "https://example.com/social.org"
|
||||||
|
self.feed = Feed.objects.create(url=self.feed_url)
|
||||||
|
|
||||||
|
def _parsed_feed(self, content):
|
||||||
|
return {
|
||||||
|
"metadata": {"title": "Alice", "nick": "alice"},
|
||||||
|
"posts": [
|
||||||
|
{
|
||||||
|
"id": "2026-01-01T10:00:00+00:00",
|
||||||
|
"content": content,
|
||||||
|
"properties": {},
|
||||||
|
"mentions": [],
|
||||||
|
"poll_options": [],
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
@patch("app.feeds.parser.parse_org_social")
|
||||||
|
def test_new_post_queues_webmention_once(self, mock_parse):
|
||||||
|
from app.feeds.tasks import scan_feeds
|
||||||
|
|
||||||
|
# Given: A feed with a new post linking to an article
|
||||||
|
mock_parse.return_value = self._parsed_feed(
|
||||||
|
"Article: https://blog.example.org/post"
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The feed is scanned twice without changes
|
||||||
|
scan_feeds.call_local()
|
||||||
|
first_scan_count = OutgoingWebmention.objects.count()
|
||||||
|
scan_feeds.call_local()
|
||||||
|
|
||||||
|
# Then: Exactly one webmention is queued, the rescan adds nothing
|
||||||
|
self.assertEqual(first_scan_count, 1)
|
||||||
|
self.assertEqual(OutgoingWebmention.objects.count(), 1)
|
||||||
|
|
||||||
|
@patch("app.feeds.parser.parse_org_social")
|
||||||
|
def test_edited_post_queues_only_added_link(self, mock_parse):
|
||||||
|
from app.feeds.tasks import scan_feeds
|
||||||
|
|
||||||
|
# Given: A scanned feed whose post links to one article
|
||||||
|
mock_parse.return_value = self._parsed_feed(
|
||||||
|
"Article: https://blog.example.org/a"
|
||||||
|
)
|
||||||
|
scan_feeds.call_local()
|
||||||
|
self.assertEqual(OutgoingWebmention.objects.count(), 1)
|
||||||
|
|
||||||
|
# When: The post is edited adding a second link and rescanned
|
||||||
|
mock_parse.return_value = self._parsed_feed(
|
||||||
|
"Article: https://blog.example.org/a and https://blog.example.org/b"
|
||||||
|
)
|
||||||
|
scan_feeds.call_local()
|
||||||
|
|
||||||
|
# Then: Only the added link produced a new webmention
|
||||||
|
self.assertEqual(OutgoingWebmention.objects.count(), 2)
|
||||||
|
targets = set(OutgoingWebmention.objects.values_list("target", flat=True))
|
||||||
|
self.assertEqual(
|
||||||
|
targets, {"https://blog.example.org/a", "https://blog.example.org/b"}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class SendPendingWebmentionsTest(TestCase):
|
||||||
|
"""Test cases for the delivery task."""
|
||||||
|
|
||||||
|
def _create_webmention(self, **kwargs):
|
||||||
|
defaults = {
|
||||||
|
"source": "https://example.com/social.org#2026-01-01T10:00:00+00:00",
|
||||||
|
"target": "https://blog.example.org/post",
|
||||||
|
}
|
||||||
|
defaults.update(kwargs)
|
||||||
|
return OutgoingWebmention.objects.create(**defaults)
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.is_safe_endpoint", return_value=True)
|
||||||
|
@patch("app.feeds.webmentions.send_webmention")
|
||||||
|
@patch("app.feeds.webmentions.discover_webmention_endpoint")
|
||||||
|
def test_pending_webmention_is_sent(self, mock_discover, mock_send, _mock_safe):
|
||||||
|
# Given: A pending webmention whose target accepts it with 202
|
||||||
|
mock_discover.return_value = "https://blog.example.org/wm"
|
||||||
|
mock_send.return_value = 202
|
||||||
|
webmention = self._create_webmention()
|
||||||
|
|
||||||
|
# When: The delivery task runs
|
||||||
|
counters = _send_pending_webmentions_impl()
|
||||||
|
|
||||||
|
# Then: The webmention is delivered once and marked as sent
|
||||||
|
self.assertEqual(counters["sent"], 1)
|
||||||
|
webmention.refresh_from_db()
|
||||||
|
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_SENT)
|
||||||
|
self.assertEqual(webmention.endpoint, "https://blog.example.org/wm")
|
||||||
|
self.assertEqual(webmention.response_code, 202)
|
||||||
|
self.assertEqual(webmention.attempts, 1)
|
||||||
|
mock_send.assert_called_once_with(
|
||||||
|
"https://blog.example.org/wm", webmention.source, webmention.target
|
||||||
|
)
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.discover_webmention_endpoint")
|
||||||
|
def test_target_without_endpoint_is_marked_permanently(self, mock_discover):
|
||||||
|
# Given: A pending webmention whose target has no endpoint
|
||||||
|
mock_discover.return_value = None
|
||||||
|
webmention = self._create_webmention()
|
||||||
|
|
||||||
|
# When: The delivery task runs
|
||||||
|
counters = _send_pending_webmentions_impl()
|
||||||
|
|
||||||
|
# Then: The webmention is marked permanently as no_endpoint
|
||||||
|
self.assertEqual(counters["no_endpoint"], 1)
|
||||||
|
webmention.refresh_from_db()
|
||||||
|
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_NO_ENDPOINT)
|
||||||
|
|
||||||
|
# Then: A later run never fetches that target again
|
||||||
|
mock_discover.reset_mock()
|
||||||
|
_send_pending_webmentions_impl()
|
||||||
|
mock_discover.assert_not_called()
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.is_safe_endpoint", return_value=False)
|
||||||
|
@patch("app.feeds.webmentions.discover_webmention_endpoint")
|
||||||
|
def test_unsafe_endpoint_is_rejected(self, mock_discover, _mock_safe):
|
||||||
|
# Given: A pending webmention whose endpoint points to loopback
|
||||||
|
mock_discover.return_value = "http://127.0.0.1/wm"
|
||||||
|
webmention = self._create_webmention()
|
||||||
|
|
||||||
|
# When: The delivery task runs
|
||||||
|
counters = _send_pending_webmentions_impl()
|
||||||
|
|
||||||
|
# Then: Nothing is sent and the webmention is discarded
|
||||||
|
self.assertEqual(counters["no_endpoint"], 1)
|
||||||
|
webmention.refresh_from_db()
|
||||||
|
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_NO_ENDPOINT)
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.discover_webmention_endpoint")
|
||||||
|
def test_network_error_marks_failed_and_retries_later(self, mock_discover):
|
||||||
|
# Given: A pending webmention whose target is unreachable
|
||||||
|
mock_discover.side_effect = requests.RequestException("boom")
|
||||||
|
webmention = self._create_webmention()
|
||||||
|
|
||||||
|
# When: The delivery task runs
|
||||||
|
counters = _send_pending_webmentions_impl()
|
||||||
|
|
||||||
|
# Then: The webmention is marked failed with one attempt
|
||||||
|
self.assertEqual(counters["failed"], 1)
|
||||||
|
webmention.refresh_from_db()
|
||||||
|
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_FAILED)
|
||||||
|
self.assertEqual(webmention.attempts, 1)
|
||||||
|
|
||||||
|
# Then: Within the backoff window it is skipped
|
||||||
|
mock_discover.reset_mock()
|
||||||
|
_send_pending_webmentions_impl()
|
||||||
|
mock_discover.assert_not_called()
|
||||||
|
|
||||||
|
# Then: After the backoff window it is retried
|
||||||
|
webmention.last_attempt_at = timezone.now() - timedelta(hours=2)
|
||||||
|
webmention.save()
|
||||||
|
mock_discover.side_effect = None
|
||||||
|
mock_discover.return_value = None
|
||||||
|
_send_pending_webmentions_impl()
|
||||||
|
webmention.refresh_from_db()
|
||||||
|
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_NO_ENDPOINT)
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.discover_webmention_endpoint")
|
||||||
|
def test_max_attempts_reached_stops_retrying(self, mock_discover):
|
||||||
|
# Given: A webmention that already exhausted its attempts long ago
|
||||||
|
self._create_webmention(
|
||||||
|
status=OutgoingWebmention.STATUS_FAILED,
|
||||||
|
attempts=WEBMENTION_MAX_ATTEMPTS,
|
||||||
|
last_attempt_at=timezone.now() - timedelta(days=30),
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: The delivery task runs
|
||||||
|
counters = _send_pending_webmentions_impl()
|
||||||
|
|
||||||
|
# Then: It is never processed again
|
||||||
|
self.assertEqual(counters, {"sent": 0, "no_endpoint": 0, "failed": 0})
|
||||||
|
mock_discover.assert_not_called()
|
||||||
|
|
||||||
|
@patch("app.feeds.webmentions.is_safe_endpoint", return_value=True)
|
||||||
|
@patch("app.feeds.webmentions.send_webmention")
|
||||||
|
@patch("app.feeds.webmentions.discover_webmention_endpoint")
|
||||||
|
def test_http_error_response_marks_failed(
|
||||||
|
self, mock_discover, mock_send, _mock_safe
|
||||||
|
):
|
||||||
|
# Given: A pending webmention whose endpoint answers HTTP 500
|
||||||
|
mock_discover.return_value = "https://blog.example.org/wm"
|
||||||
|
mock_send.return_value = 500
|
||||||
|
webmention = self._create_webmention()
|
||||||
|
|
||||||
|
# When: The delivery task runs
|
||||||
|
counters = _send_pending_webmentions_impl()
|
||||||
|
|
||||||
|
# Then: The webmention is marked failed keeping the endpoint for retry
|
||||||
|
self.assertEqual(counters["failed"], 1)
|
||||||
|
webmention.refresh_from_db()
|
||||||
|
self.assertEqual(webmention.status, OutgoingWebmention.STATUS_FAILED)
|
||||||
|
self.assertEqual(webmention.response_code, 500)
|
||||||
|
self.assertEqual(webmention.endpoint, "https://blog.example.org/wm")
|
||||||
|
|
@ -4,6 +4,8 @@ from rest_framework import status
|
||||||
from django.core.cache import cache
|
from django.core.cache import cache
|
||||||
import logging
|
import logging
|
||||||
|
|
||||||
|
from app.bridge.models import bridge_urls_q, is_bridge_feed_url
|
||||||
|
|
||||||
from .models import Feed
|
from .models import Feed
|
||||||
from .parser import validate_org_social_feed
|
from .parser import validate_org_social_feed
|
||||||
|
|
||||||
|
|
@ -32,8 +34,11 @@ class FeedsView(APIView):
|
||||||
status=status.HTTP_200_OK,
|
status=status.HTTP_200_OK,
|
||||||
)
|
)
|
||||||
|
|
||||||
# If not in cache, query database
|
# If not in cache, query database. Bridge virtual feeds are a
|
||||||
feeds = list(Feed.objects.all().values_list("url", flat=True))
|
# connection helper, not real accounts, so they are never listed.
|
||||||
|
feeds = list(
|
||||||
|
Feed.objects.exclude(bridge_urls_q("url")).values_list("url", flat=True)
|
||||||
|
)
|
||||||
|
|
||||||
# Cache permanently (will be cleared by scan_feeds task)
|
# Cache permanently (will be cleared by scan_feeds task)
|
||||||
cache.set(cache_key, feeds, None)
|
cache.set(cache_key, feeds, None)
|
||||||
|
|
@ -62,6 +67,18 @@ class FeedsView(APIView):
|
||||||
|
|
||||||
feed_url = feed_url.strip()
|
feed_url = feed_url.strip()
|
||||||
|
|
||||||
|
# Bridge virtual feeds serve valid Org Social content but are not
|
||||||
|
# real accounts; they cannot be registered
|
||||||
|
if is_bridge_feed_url(feed_url):
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Error",
|
||||||
|
"errors": ["Bridge virtual feeds cannot be registered"],
|
||||||
|
"data": None,
|
||||||
|
},
|
||||||
|
status=status.HTTP_400_BAD_REQUEST,
|
||||||
|
)
|
||||||
|
|
||||||
# Check if feed already exists
|
# Check if feed already exists
|
||||||
existing_feed = Feed.objects.filter(url=feed_url).first()
|
existing_feed = Feed.objects.filter(url=feed_url).first()
|
||||||
if existing_feed:
|
if existing_feed:
|
||||||
|
|
|
||||||
202
app/feeds/webmentions.py
Normal file
202
app/feeds/webmentions.py
Normal file
|
|
@ -0,0 +1,202 @@
|
||||||
|
"""
|
||||||
|
Outgoing Webmention support (sender side only).
|
||||||
|
|
||||||
|
Implements the sender half of https://www.w3.org/TR/webmention/:
|
||||||
|
URL extraction from post content, endpoint discovery and notification.
|
||||||
|
Receiving webmentions is out of scope for the relay.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import ipaddress
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
import socket
|
||||||
|
from html.parser import HTMLParser
|
||||||
|
from urllib.parse import urljoin, urlparse
|
||||||
|
|
||||||
|
import requests
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
WEBMENTION_TIMEOUT = 10
|
||||||
|
WEBMENTION_USER_AGENT = "Org-Social-Relay (+https://relay.org-social.org/)"
|
||||||
|
# Maximum HTML bytes read while looking for a <link>/<a> endpoint
|
||||||
|
DISCOVERY_MAX_BYTES = 1024 * 1024
|
||||||
|
|
||||||
|
# Org Social mentions ([[org-social:url][nick]]) are handled natively by the
|
||||||
|
# relay and must never produce webmentions
|
||||||
|
ORG_SOCIAL_MENTION_RE = re.compile(r"\[\[org-social:[^\]]*\](?:\[[^\]]*\])?\]")
|
||||||
|
ORG_LINK_RE = re.compile(r"\[\[(https?://[^\]\[]+)\](?:\[[^\]]*\])?\]")
|
||||||
|
BARE_URL_RE = re.compile(r"https?://[^\s\]\[<>\"']+")
|
||||||
|
TRAILING_PUNCTUATION = ".,;:!?)'\""
|
||||||
|
|
||||||
|
|
||||||
|
def extract_external_urls(content):
|
||||||
|
"""
|
||||||
|
Extract http(s) URLs from post content, both Org links ([[url]] and
|
||||||
|
[[url][description]]) and bare URLs. Org Social mentions are ignored.
|
||||||
|
|
||||||
|
Returns a list of unique URLs in order of appearance.
|
||||||
|
"""
|
||||||
|
if not content:
|
||||||
|
return []
|
||||||
|
|
||||||
|
text = ORG_SOCIAL_MENTION_RE.sub(" ", content)
|
||||||
|
|
||||||
|
urls = []
|
||||||
|
for match in ORG_LINK_RE.finditer(text):
|
||||||
|
urls.append(match.group(1).strip())
|
||||||
|
|
||||||
|
# Remove Org link syntax so bare URL matching does not see them again
|
||||||
|
text = ORG_LINK_RE.sub(" ", text)
|
||||||
|
|
||||||
|
for match in BARE_URL_RE.finditer(text):
|
||||||
|
urls.append(match.group(0).rstrip(TRAILING_PUNCTUATION))
|
||||||
|
|
||||||
|
unique_urls = []
|
||||||
|
seen = set()
|
||||||
|
for url in urls:
|
||||||
|
if url and url not in seen:
|
||||||
|
seen.add(url)
|
||||||
|
unique_urls.append(url)
|
||||||
|
return unique_urls
|
||||||
|
|
||||||
|
|
||||||
|
class _EndpointHTMLParser(HTMLParser):
|
||||||
|
"""Finds the first <link> or <a> with rel="webmention" in document order."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
super().__init__(convert_charrefs=True)
|
||||||
|
self.endpoint = None
|
||||||
|
|
||||||
|
def handle_starttag(self, tag, attrs):
|
||||||
|
if self.endpoint is not None or tag not in ("link", "a"):
|
||||||
|
return
|
||||||
|
attrs_dict = dict(attrs)
|
||||||
|
rel = attrs_dict.get("rel") or ""
|
||||||
|
if "webmention" in rel.lower().split() and "href" in attrs_dict:
|
||||||
|
# An empty href is valid: it resolves to the page URL itself
|
||||||
|
self.endpoint = attrs_dict.get("href") or ""
|
||||||
|
|
||||||
|
|
||||||
|
def discover_webmention_endpoint(target_url):
|
||||||
|
"""
|
||||||
|
Discover the Webmention endpoint of a target URL following the spec
|
||||||
|
precedence: first HTTP Link header, then first <link>/<a> element in
|
||||||
|
document order. Relative endpoints are resolved against the final URL
|
||||||
|
after redirects.
|
||||||
|
|
||||||
|
Returns the absolute endpoint URL, or None if the target does not
|
||||||
|
advertise one. Raises requests.RequestException on network errors.
|
||||||
|
"""
|
||||||
|
response = requests.get(
|
||||||
|
target_url,
|
||||||
|
timeout=WEBMENTION_TIMEOUT,
|
||||||
|
headers={"User-Agent": WEBMENTION_USER_AGENT},
|
||||||
|
stream=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
link_header = response.headers.get("Link", "")
|
||||||
|
if link_header:
|
||||||
|
for link in requests.utils.parse_header_links(link_header):
|
||||||
|
rels = link.get("rel", "").lower().split()
|
||||||
|
if "webmention" in rels and "url" in link:
|
||||||
|
return urljoin(response.url, link["url"])
|
||||||
|
|
||||||
|
content_type = response.headers.get("Content-Type", "")
|
||||||
|
if "html" not in content_type.lower():
|
||||||
|
return None
|
||||||
|
|
||||||
|
raw_content = response.raw.read(DISCOVERY_MAX_BYTES, decode_content=True)
|
||||||
|
html = raw_content.decode(response.encoding or "utf-8", errors="replace")
|
||||||
|
finally:
|
||||||
|
response.close()
|
||||||
|
|
||||||
|
parser = _EndpointHTMLParser()
|
||||||
|
parser.feed(html)
|
||||||
|
|
||||||
|
if parser.endpoint is None:
|
||||||
|
return None
|
||||||
|
return urljoin(response.url, parser.endpoint)
|
||||||
|
|
||||||
|
|
||||||
|
def is_safe_endpoint(endpoint_url):
|
||||||
|
"""
|
||||||
|
Reject endpoints that are not plain http(s) or that resolve to loopback,
|
||||||
|
private, link-local or otherwise non-global addresses (spec section 4.3).
|
||||||
|
"""
|
||||||
|
parsed = urlparse(endpoint_url)
|
||||||
|
if parsed.scheme not in ("http", "https") or not parsed.hostname:
|
||||||
|
return False
|
||||||
|
|
||||||
|
try:
|
||||||
|
addr_info = socket.getaddrinfo(parsed.hostname, None)
|
||||||
|
except (socket.gaierror, UnicodeError):
|
||||||
|
return False
|
||||||
|
|
||||||
|
for info in addr_info:
|
||||||
|
try:
|
||||||
|
address = ipaddress.ip_address(info[4][0])
|
||||||
|
except ValueError:
|
||||||
|
return False
|
||||||
|
if (
|
||||||
|
address.is_loopback
|
||||||
|
or address.is_private
|
||||||
|
or address.is_link_local
|
||||||
|
or address.is_multicast
|
||||||
|
or address.is_reserved
|
||||||
|
or address.is_unspecified
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def send_webmention(endpoint_url, source, target):
|
||||||
|
"""
|
||||||
|
POST the webmention as application/x-www-form-urlencoded and return the
|
||||||
|
HTTP status code. Raises requests.RequestException on network errors.
|
||||||
|
"""
|
||||||
|
response = requests.post(
|
||||||
|
endpoint_url,
|
||||||
|
data={"source": source, "target": target},
|
||||||
|
timeout=WEBMENTION_TIMEOUT,
|
||||||
|
headers={"User-Agent": WEBMENTION_USER_AGENT},
|
||||||
|
)
|
||||||
|
return response.status_code
|
||||||
|
|
||||||
|
|
||||||
|
def queue_webmentions_for_post(profile, post_id, content):
|
||||||
|
"""
|
||||||
|
Create pending OutgoingWebmention rows for every external URL in a post.
|
||||||
|
|
||||||
|
URLs pointing to registered Org Social feeds (or the author's own feed)
|
||||||
|
are skipped: the relay already handles those interactions natively.
|
||||||
|
Existing (source, target) pairs are never touched, so a webmention is
|
||||||
|
sent at most once even across rescans and post edits.
|
||||||
|
|
||||||
|
Returns the number of newly queued webmentions.
|
||||||
|
"""
|
||||||
|
from .models import Feed, OutgoingWebmention
|
||||||
|
|
||||||
|
urls = extract_external_urls(content)
|
||||||
|
if not urls:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
source = f"{profile.feed}#{post_id}"
|
||||||
|
base_urls = {url.split("#")[0] for url in urls}
|
||||||
|
known_feeds = set(
|
||||||
|
Feed.objects.filter(url__in=base_urls).values_list("url", flat=True)
|
||||||
|
)
|
||||||
|
|
||||||
|
queued = 0
|
||||||
|
for url in urls:
|
||||||
|
base_url = url.split("#")[0]
|
||||||
|
if base_url == profile.feed or base_url in known_feeds:
|
||||||
|
continue
|
||||||
|
_, created = OutgoingWebmention.objects.get_or_create(source=source, target=url)
|
||||||
|
if created:
|
||||||
|
queued += 1
|
||||||
|
|
||||||
|
if queued:
|
||||||
|
logger.info(f"Queued {queued} webmention(s) for {source}")
|
||||||
|
return queued
|
||||||
0
app/interactions/__init__.py
Normal file
0
app/interactions/__init__.py
Normal file
6
app/interactions/apps.py
Normal file
6
app/interactions/apps.py
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
from django.apps import AppConfig
|
||||||
|
|
||||||
|
|
||||||
|
class InteractionsConfig(AppConfig):
|
||||||
|
default_auto_field = "django.db.models.BigAutoField"
|
||||||
|
name = "app.interactions"
|
||||||
323
app/interactions/tests.py
Normal file
323
app/interactions/tests.py
Normal file
|
|
@ -0,0 +1,323 @@
|
||||||
|
from django.test import TestCase
|
||||||
|
from rest_framework.test import APIClient
|
||||||
|
from rest_framework import status
|
||||||
|
|
||||||
|
from app.feeds.models import Profile, Post
|
||||||
|
|
||||||
|
|
||||||
|
class InteractionsViewTest(TestCase):
|
||||||
|
"""Test cases for the InteractionsView API using Given/When/Then structure."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.client = APIClient()
|
||||||
|
self.interactions_url = "/interactions/"
|
||||||
|
|
||||||
|
# Create test profiles
|
||||||
|
self.profile1 = Profile.objects.create(
|
||||||
|
feed="https://example.com/social.org",
|
||||||
|
title="Example Profile",
|
||||||
|
nick="example_user",
|
||||||
|
description="Test profile 1",
|
||||||
|
)
|
||||||
|
self.profile2 = Profile.objects.create(
|
||||||
|
feed="https://alice.com/social.org",
|
||||||
|
title="Alice Profile",
|
||||||
|
nick="alice",
|
||||||
|
description="Alice's profile",
|
||||||
|
)
|
||||||
|
self.profile3 = Profile.objects.create(
|
||||||
|
feed="https://bob.com/social.org",
|
||||||
|
title="Bob Profile",
|
||||||
|
nick="bob",
|
||||||
|
description="Bob's profile",
|
||||||
|
)
|
||||||
|
self.profile4 = Profile.objects.create(
|
||||||
|
feed="https://charlie.com/social.org",
|
||||||
|
title="Charlie Profile",
|
||||||
|
nick="charlie",
|
||||||
|
description="Charlie's profile",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create original post
|
||||||
|
self.original_post = Post.objects.create(
|
||||||
|
profile=self.profile1,
|
||||||
|
post_id="2025-02-05T10:00:00+0100",
|
||||||
|
content="This is an amazing discovery!",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create reactions
|
||||||
|
self.reaction1 = Post.objects.create(
|
||||||
|
profile=self.profile2,
|
||||||
|
post_id="2025-02-05T13:15:00+0100",
|
||||||
|
content="",
|
||||||
|
mood="❤",
|
||||||
|
reply_to=f"{self.profile1.feed}#{self.original_post.post_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.reaction2 = Post.objects.create(
|
||||||
|
profile=self.profile3,
|
||||||
|
post_id="2025-02-05T14:30:00+0100",
|
||||||
|
content="",
|
||||||
|
mood="🚀",
|
||||||
|
reply_to=f"{self.profile1.feed}#{self.original_post.post_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create replies
|
||||||
|
self.reply1 = Post.objects.create(
|
||||||
|
profile=self.profile4,
|
||||||
|
post_id="2025-02-05T12:30:00+0100",
|
||||||
|
content="Great post!",
|
||||||
|
mood="",
|
||||||
|
reply_to=f"{self.profile1.feed}#{self.original_post.post_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.reply2 = Post.objects.create(
|
||||||
|
profile=self.profile2,
|
||||||
|
post_id="2025-02-05T15:00:00+0100",
|
||||||
|
content="I agree!",
|
||||||
|
mood="",
|
||||||
|
reply_to=f"{self.profile1.feed}#{self.original_post.post_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create boosts
|
||||||
|
self.boost1 = Post.objects.create(
|
||||||
|
profile=self.profile2,
|
||||||
|
post_id="2025-02-05T14:00:00+0100",
|
||||||
|
content="Guys, you have to see this!",
|
||||||
|
include=f"{self.profile1.feed}#{self.original_post.post_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.boost2 = Post.objects.create(
|
||||||
|
profile=self.profile3,
|
||||||
|
post_id="2025-02-05T15:30:00+0100",
|
||||||
|
content="",
|
||||||
|
include=f"{self.profile1.feed}#{self.original_post.post_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_get_interactions_success(self):
|
||||||
|
"""Test GET /interactions/?post=<post_url> returns all interactions."""
|
||||||
|
# Given: A post with reactions, replies, and boosts
|
||||||
|
post_url = f"{self.profile1.feed}#{self.original_post.post_id}"
|
||||||
|
|
||||||
|
# When: We request interactions for the post
|
||||||
|
response = self.client.get(self.interactions_url, {"post": post_url})
|
||||||
|
|
||||||
|
# Then: We should get all interactions successfully
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
self.assertEqual(response.data["type"], "Success")
|
||||||
|
self.assertEqual(response.data["errors"], [])
|
||||||
|
|
||||||
|
# Then: Response should contain all three types
|
||||||
|
data = response.data["data"]
|
||||||
|
self.assertIn("reactions", data)
|
||||||
|
self.assertIn("replies", data)
|
||||||
|
self.assertIn("boosts", data)
|
||||||
|
|
||||||
|
# Verify reactions
|
||||||
|
reactions = data["reactions"]
|
||||||
|
self.assertEqual(len(reactions), 2)
|
||||||
|
self.assertEqual(reactions[0]["emoji"], "🚀") # Most recent first
|
||||||
|
self.assertEqual(reactions[1]["emoji"], "❤")
|
||||||
|
|
||||||
|
# Verify replies
|
||||||
|
replies = data["replies"]
|
||||||
|
self.assertEqual(len(replies), 2)
|
||||||
|
self.assertIn(f"{self.profile2.feed}#{self.reply2.post_id}", replies)
|
||||||
|
self.assertIn(f"{self.profile4.feed}#{self.reply1.post_id}", replies)
|
||||||
|
|
||||||
|
# Verify boosts
|
||||||
|
boosts = data["boosts"]
|
||||||
|
self.assertEqual(len(boosts), 2)
|
||||||
|
self.assertIn(f"{self.profile2.feed}#{self.boost1.post_id}", boosts)
|
||||||
|
self.assertIn(f"{self.profile3.feed}#{self.boost2.post_id}", boosts)
|
||||||
|
|
||||||
|
# Verify meta
|
||||||
|
meta = response.data["meta"]
|
||||||
|
self.assertEqual(meta["post"], post_url)
|
||||||
|
self.assertEqual(meta["total_reactions"], 2)
|
||||||
|
self.assertEqual(meta["total_replies"], 2)
|
||||||
|
self.assertEqual(meta["total_boosts"], 2)
|
||||||
|
self.assertEqual(meta["parentChain"], []) # Original post has no parents
|
||||||
|
|
||||||
|
# Verify links
|
||||||
|
links = response.data["_links"]
|
||||||
|
self.assertIn("self", links)
|
||||||
|
self.assertIn("reactions", links)
|
||||||
|
self.assertIn("replies", links)
|
||||||
|
self.assertIn("boosts", links)
|
||||||
|
|
||||||
|
def test_get_interactions_with_parent_chain(self):
|
||||||
|
"""Test GET /interactions/ includes parent chain for reply posts."""
|
||||||
|
# Given: A post that is a reply (has parents)
|
||||||
|
parent_post = Post.objects.create(
|
||||||
|
profile=self.profile2,
|
||||||
|
post_id="2025-02-05T08:00:00+0100",
|
||||||
|
content="Parent post",
|
||||||
|
)
|
||||||
|
|
||||||
|
child_post = Post.objects.create(
|
||||||
|
profile=self.profile1,
|
||||||
|
post_id="2025-02-05T09:00:00+0100",
|
||||||
|
content="Reply to parent",
|
||||||
|
reply_to=f"{self.profile2.feed}#{parent_post.post_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
post_url = f"{self.profile1.feed}#{child_post.post_id}"
|
||||||
|
|
||||||
|
# When: We request interactions for the child post
|
||||||
|
response = self.client.get(self.interactions_url, {"post": post_url})
|
||||||
|
|
||||||
|
# Then: Should include parent chain
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
meta = response.data["meta"]
|
||||||
|
self.assertIn("parentChain", meta)
|
||||||
|
self.assertEqual(len(meta["parentChain"]), 1)
|
||||||
|
self.assertEqual(
|
||||||
|
meta["parentChain"][0], f"{self.profile2.feed}#{parent_post.post_id}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_get_interactions_no_interactions(self):
|
||||||
|
"""Test GET /interactions/ when post has no interactions."""
|
||||||
|
# Given: A post without any interactions
|
||||||
|
lonely_post = Post.objects.create(
|
||||||
|
profile=self.profile1,
|
||||||
|
post_id="2025-02-05T20:00:00+0100",
|
||||||
|
content="Lonely post",
|
||||||
|
)
|
||||||
|
post_url = f"{self.profile1.feed}#{lonely_post.post_id}"
|
||||||
|
|
||||||
|
# When: We request interactions
|
||||||
|
response = self.client.get(self.interactions_url, {"post": post_url})
|
||||||
|
|
||||||
|
# Then: Should return empty arrays
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
data = response.data["data"]
|
||||||
|
self.assertEqual(len(data["reactions"]), 0)
|
||||||
|
self.assertEqual(len(data["replies"]), 0)
|
||||||
|
self.assertEqual(len(data["boosts"]), 0)
|
||||||
|
|
||||||
|
meta = response.data["meta"]
|
||||||
|
self.assertEqual(meta["total_reactions"], 0)
|
||||||
|
self.assertEqual(meta["total_replies"], 0)
|
||||||
|
self.assertEqual(meta["total_boosts"], 0)
|
||||||
|
|
||||||
|
def test_get_interactions_missing_post_parameter(self):
|
||||||
|
"""Test GET /interactions/ without post parameter returns error."""
|
||||||
|
# Given: No post parameter provided
|
||||||
|
|
||||||
|
# When: We request interactions without post parameter
|
||||||
|
response = self.client.get(self.interactions_url)
|
||||||
|
|
||||||
|
# Then: We should get a 400 error
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn("'post' parameter is required", response.data["errors"])
|
||||||
|
|
||||||
|
def test_get_interactions_invalid_post_url_format(self):
|
||||||
|
"""Test GET /interactions/?post=<invalid_url> returns error."""
|
||||||
|
# Given: An invalid post URL (missing #)
|
||||||
|
invalid_url = "https://example.com/social.org"
|
||||||
|
|
||||||
|
# When: We request interactions with invalid URL
|
||||||
|
response = self.client.get(self.interactions_url, {"post": invalid_url})
|
||||||
|
|
||||||
|
# Then: We should get a 400 error
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn("Invalid post URL format", response.data["errors"][0])
|
||||||
|
|
||||||
|
def test_get_interactions_post_not_found(self):
|
||||||
|
"""Test GET /interactions/?post=<nonexistent_post> returns 404."""
|
||||||
|
# Given: A non-existent post URL
|
||||||
|
post_url = "https://nonexistent.com/social.org#2025-01-01T00:00:00+00:00"
|
||||||
|
|
||||||
|
# When: We request interactions for non-existent post
|
||||||
|
response = self.client.get(self.interactions_url, {"post": post_url})
|
||||||
|
|
||||||
|
# Then: We should get a 404 error
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn("Post not found", response.data["errors"])
|
||||||
|
|
||||||
|
def test_interactions_excludes_poll_votes(self):
|
||||||
|
"""Test that poll votes are excluded from reactions and replies."""
|
||||||
|
# Given: A post with a poll vote (should be excluded)
|
||||||
|
poll_post = Post.objects.create(
|
||||||
|
profile=self.profile2,
|
||||||
|
post_id="2025-02-05T16:00:00+0100",
|
||||||
|
content="Poll post",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create a poll vote (reply with poll_votes relationship)
|
||||||
|
vote_post = Post.objects.create(
|
||||||
|
profile=self.profile3,
|
||||||
|
post_id="2025-02-05T17:00:00+0100",
|
||||||
|
content="",
|
||||||
|
reply_to=f"{self.profile1.feed}#{self.original_post.post_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
from app.feeds.models import PollVote
|
||||||
|
|
||||||
|
PollVote.objects.create(
|
||||||
|
post=vote_post, poll_post=poll_post, poll_option="Option A"
|
||||||
|
)
|
||||||
|
|
||||||
|
post_url = f"{self.profile1.feed}#{self.original_post.post_id}"
|
||||||
|
|
||||||
|
# When: We request interactions
|
||||||
|
response = self.client.get(self.interactions_url, {"post": post_url})
|
||||||
|
|
||||||
|
# Then: Poll vote should not be in replies or reactions
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
vote_url = f"{self.profile3.feed}#{vote_post.post_id}"
|
||||||
|
|
||||||
|
data = response.data["data"]
|
||||||
|
self.assertNotIn(vote_url, data["replies"])
|
||||||
|
for reaction in data["reactions"]:
|
||||||
|
self.assertNotEqual(reaction["post"], vote_url)
|
||||||
|
|
||||||
|
def test_interactions_caching(self):
|
||||||
|
"""Test that interactions responses are cached."""
|
||||||
|
# Given: A post with interactions
|
||||||
|
post_url = f"{self.profile1.feed}#{self.original_post.post_id}"
|
||||||
|
|
||||||
|
# When: We request interactions twice
|
||||||
|
response1 = self.client.get(self.interactions_url, {"post": post_url})
|
||||||
|
response2 = self.client.get(self.interactions_url, {"post": post_url})
|
||||||
|
|
||||||
|
# Then: Both responses should be identical
|
||||||
|
self.assertEqual(response1.data, response2.data)
|
||||||
|
self.assertEqual(response1.status_code, status.HTTP_200_OK)
|
||||||
|
self.assertEqual(response2.status_code, status.HTTP_200_OK)
|
||||||
|
|
||||||
|
def test_interactions_ordered_by_recency(self):
|
||||||
|
"""Test that all interactions are ordered from most recent to oldest."""
|
||||||
|
# Given: A post with interactions
|
||||||
|
post_url = f"{self.profile1.feed}#{self.original_post.post_id}"
|
||||||
|
|
||||||
|
# When: We request interactions
|
||||||
|
response = self.client.get(self.interactions_url, {"post": post_url})
|
||||||
|
|
||||||
|
# Then: Items should be ordered by post_id descending (most recent first)
|
||||||
|
data = response.data["data"]
|
||||||
|
|
||||||
|
# Check reactions order
|
||||||
|
if len(data["reactions"]) > 1:
|
||||||
|
for i in range(len(data["reactions"]) - 1):
|
||||||
|
current_id = data["reactions"][i]["post"].split("#")[1]
|
||||||
|
next_id = data["reactions"][i + 1]["post"].split("#")[1]
|
||||||
|
self.assertGreater(current_id, next_id)
|
||||||
|
|
||||||
|
# Check replies order
|
||||||
|
if len(data["replies"]) > 1:
|
||||||
|
for i in range(len(data["replies"]) - 1):
|
||||||
|
current_id = data["replies"][i].split("#")[1]
|
||||||
|
next_id = data["replies"][i + 1].split("#")[1]
|
||||||
|
self.assertGreater(current_id, next_id)
|
||||||
|
|
||||||
|
# Check boosts order
|
||||||
|
if len(data["boosts"]) > 1:
|
||||||
|
for i in range(len(data["boosts"]) - 1):
|
||||||
|
current_id = data["boosts"][i].split("#")[1]
|
||||||
|
next_id = data["boosts"][i + 1].split("#")[1]
|
||||||
|
self.assertGreater(current_id, next_id)
|
||||||
8
app/interactions/urls.py
Normal file
8
app/interactions/urls.py
Normal file
|
|
@ -0,0 +1,8 @@
|
||||||
|
from django.urls import path
|
||||||
|
from .views import InteractionsView
|
||||||
|
|
||||||
|
app_name = "interactions"
|
||||||
|
|
||||||
|
urlpatterns = [
|
||||||
|
path("", InteractionsView.as_view(), name="interactions"),
|
||||||
|
]
|
||||||
134
app/interactions/views.py
Normal file
134
app/interactions/views.py
Normal file
|
|
@ -0,0 +1,134 @@
|
||||||
|
from rest_framework.views import APIView
|
||||||
|
from rest_framework.response import Response
|
||||||
|
from rest_framework import status
|
||||||
|
from django.core.cache import cache
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from app.feeds.models import Post, Profile
|
||||||
|
from app.feeds.utils import get_parent_chain
|
||||||
|
from app.reactions.utils import get_reactions_for_post
|
||||||
|
from app.replies.utils import get_direct_replies_for_post
|
||||||
|
from app.boosts.utils import get_boosts_for_post
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class InteractionsView(APIView):
|
||||||
|
"""Get all interactions (reactions, replies, boosts) for a specific post"""
|
||||||
|
|
||||||
|
def get(self, request):
|
||||||
|
post_url = request.query_params.get("post")
|
||||||
|
|
||||||
|
if not post_url:
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Error",
|
||||||
|
"errors": ["'post' parameter is required"],
|
||||||
|
"data": None,
|
||||||
|
},
|
||||||
|
status=status.HTTP_400_BAD_REQUEST,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Parse the post URL format: https://feed.com/social.org#post_id
|
||||||
|
try:
|
||||||
|
if "#" not in post_url:
|
||||||
|
raise ValueError("Invalid post URL format")
|
||||||
|
feed_url, post_id = post_url.split("#", 1)
|
||||||
|
except ValueError:
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Error",
|
||||||
|
"errors": ["Invalid post URL format. Expected: feed_url#post_id"],
|
||||||
|
"data": None,
|
||||||
|
},
|
||||||
|
status=status.HTTP_400_BAD_REQUEST,
|
||||||
|
)
|
||||||
|
|
||||||
|
cache_key = f"interactions_{feed_url}_{post_id}"
|
||||||
|
cached_response = cache.get(cache_key)
|
||||||
|
|
||||||
|
if cached_response is not None:
|
||||||
|
return Response(cached_response, status=status.HTTP_200_OK)
|
||||||
|
|
||||||
|
# Find the original post
|
||||||
|
try:
|
||||||
|
profile = Profile.objects.get(feed=feed_url)
|
||||||
|
original_post = Post.objects.get(profile=profile, post_id=post_id)
|
||||||
|
except (Profile.DoesNotExist, Post.DoesNotExist):
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Error",
|
||||||
|
"errors": ["Post not found"],
|
||||||
|
"data": None,
|
||||||
|
},
|
||||||
|
status=status.HTTP_404_NOT_FOUND,
|
||||||
|
)
|
||||||
|
|
||||||
|
original_post_url = f"{feed_url}#{post_id}"
|
||||||
|
|
||||||
|
# Get reactions using shared utility function
|
||||||
|
reactions_qs = get_reactions_for_post(original_post_url)
|
||||||
|
reactions_data = [
|
||||||
|
{
|
||||||
|
"post": f"{reaction.profile.feed}#{reaction.post_id}",
|
||||||
|
"emoji": reaction.mood,
|
||||||
|
}
|
||||||
|
for reaction in reactions_qs
|
||||||
|
]
|
||||||
|
|
||||||
|
# Get replies using shared utility function
|
||||||
|
replies_qs = get_direct_replies_for_post(original_post_url)
|
||||||
|
replies_data = [f"{reply.profile.feed}#{reply.post_id}" for reply in replies_qs]
|
||||||
|
|
||||||
|
# Get boosts using shared utility function
|
||||||
|
boosts_qs = get_boosts_for_post(original_post_url)
|
||||||
|
boosts_data = [f"{boost.profile.feed}#{boost.post_id}" for boost in boosts_qs]
|
||||||
|
|
||||||
|
# Get parent chain
|
||||||
|
parent_chain = get_parent_chain(original_post)
|
||||||
|
|
||||||
|
# URL encode the post_url for the self link
|
||||||
|
from urllib.parse import quote
|
||||||
|
|
||||||
|
encoded_post_url = quote(post_url, safe="")
|
||||||
|
encoded_feed_url = quote(feed_url, safe="")
|
||||||
|
|
||||||
|
response_data = {
|
||||||
|
"type": "Success",
|
||||||
|
"errors": [],
|
||||||
|
"data": {
|
||||||
|
"reactions": reactions_data,
|
||||||
|
"replies": replies_data,
|
||||||
|
"boosts": boosts_data,
|
||||||
|
},
|
||||||
|
"meta": {
|
||||||
|
"post": original_post_url,
|
||||||
|
"total_reactions": len(reactions_data),
|
||||||
|
"total_replies": len(replies_data),
|
||||||
|
"total_boosts": len(boosts_data),
|
||||||
|
"parentChain": parent_chain,
|
||||||
|
},
|
||||||
|
"_links": {
|
||||||
|
"self": {
|
||||||
|
"href": f"/interactions/?post={encoded_post_url}",
|
||||||
|
"method": "GET",
|
||||||
|
},
|
||||||
|
"reactions": {
|
||||||
|
"href": f"/reactions/?feed={encoded_feed_url}",
|
||||||
|
"method": "GET",
|
||||||
|
},
|
||||||
|
"replies": {
|
||||||
|
"href": f"/replies/?post={encoded_post_url}",
|
||||||
|
"method": "GET",
|
||||||
|
},
|
||||||
|
"boosts": {
|
||||||
|
"href": f"/boosts/?post={encoded_post_url}",
|
||||||
|
"method": "GET",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
# Cache permanently (will be cleared by scan_feeds task)
|
||||||
|
cache.set(cache_key, response_data, None)
|
||||||
|
|
||||||
|
return Response(response_data, status=status.HTTP_200_OK)
|
||||||
|
|
@ -346,3 +346,76 @@ class NotificationsViewTest(TestCase):
|
||||||
self.assertIn("post", reply)
|
self.assertIn("post", reply)
|
||||||
self.assertIn("parent", reply)
|
self.assertIn("parent", reply)
|
||||||
self.assertNotIn("emoji", reply)
|
self.assertNotIn("emoji", reply)
|
||||||
|
|
||||||
|
def test_get_notifications_with_boosts(self):
|
||||||
|
"""Test GET /notifications/?feed=<feed_url> includes boosts."""
|
||||||
|
# Given: A profile with posts that have been boosted
|
||||||
|
boost_post = Post.objects.create(
|
||||||
|
profile=self.profile2,
|
||||||
|
post_id="2025-01-01T17:00:00+00:00",
|
||||||
|
content="Boosting this amazing post!",
|
||||||
|
include=f"{self.profile1.feed}#{self.post1.post_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: We request all notifications for the profile
|
||||||
|
response = self.client.get(self.notifications_url, {"feed": self.profile1.feed})
|
||||||
|
|
||||||
|
# Then: We should get notifications including the boost
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
data = response.data["data"]
|
||||||
|
|
||||||
|
# Find the boost notification
|
||||||
|
boost_notifications = [n for n in data if n["type"] == "boost"]
|
||||||
|
self.assertEqual(len(boost_notifications), 1)
|
||||||
|
|
||||||
|
boost_notif = boost_notifications[0]
|
||||||
|
self.assertEqual(boost_notif["type"], "boost")
|
||||||
|
self.assertEqual(
|
||||||
|
boost_notif["post"], f"{self.profile2.feed}#{boost_post.post_id}"
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
boost_notif["boosted"], f"{self.profile1.feed}#{self.post1.post_id}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: Meta should include boost count
|
||||||
|
meta = response.data["meta"]
|
||||||
|
self.assertEqual(meta["by_type"]["boosts"], 1)
|
||||||
|
self.assertEqual(meta["total"], 4) # 1 mention + 1 reaction + 1 reply + 1 boost
|
||||||
|
|
||||||
|
def test_get_notifications_filtered_by_boost(self):
|
||||||
|
"""Test GET /notifications/?feed=<feed_url>&type=boost returns only boosts."""
|
||||||
|
# Given: A profile with all notification types including boosts
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile2,
|
||||||
|
post_id="2025-01-01T17:00:00+00:00",
|
||||||
|
content="Boosting!",
|
||||||
|
include=f"{self.profile1.feed}#{self.post1.post_id}",
|
||||||
|
)
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile3,
|
||||||
|
post_id="2025-01-01T18:00:00+00:00",
|
||||||
|
content="",
|
||||||
|
include=f"{self.profile1.feed}#{self.post2.post_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: We request only boosts
|
||||||
|
response = self.client.get(
|
||||||
|
self.notifications_url, {"feed": self.profile1.feed, "type": "boost"}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: Should only return boosts
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
data = response.data["data"]
|
||||||
|
self.assertEqual(len(data), 2)
|
||||||
|
|
||||||
|
for boost in data:
|
||||||
|
self.assertEqual(boost["type"], "boost")
|
||||||
|
self.assertIn("post", boost)
|
||||||
|
self.assertIn("boosted", boost)
|
||||||
|
self.assertNotIn("emoji", boost)
|
||||||
|
self.assertNotIn("parent", boost)
|
||||||
|
|
||||||
|
# Then: Meta should show filtered results
|
||||||
|
meta = response.data["meta"]
|
||||||
|
self.assertEqual(meta["total"], 2)
|
||||||
|
self.assertEqual(meta["by_type"]["boosts"], 2)
|
||||||
|
|
|
||||||
|
|
@ -30,7 +30,7 @@ class NotificationsView(APIView):
|
||||||
feed_url = feed_url.strip()
|
feed_url = feed_url.strip()
|
||||||
|
|
||||||
# Validate type parameter if provided
|
# Validate type parameter if provided
|
||||||
valid_types = ["mention", "reaction", "reply"]
|
valid_types = ["mention", "reaction", "reply", "boost"]
|
||||||
if notification_type and notification_type not in valid_types:
|
if notification_type and notification_type not in valid_types:
|
||||||
return Response(
|
return Response(
|
||||||
{
|
{
|
||||||
|
|
@ -64,7 +64,7 @@ class NotificationsView(APIView):
|
||||||
)
|
)
|
||||||
|
|
||||||
notifications_data = []
|
notifications_data = []
|
||||||
counts = {"mentions": 0, "reactions": 0, "replies": 0}
|
counts = {"mentions": 0, "reactions": 0, "replies": 0, "boosts": 0}
|
||||||
|
|
||||||
# Get all post IDs from this profile (for reactions and replies)
|
# Get all post IDs from this profile (for reactions and replies)
|
||||||
profile_post_ids = list(profile.posts.values_list("post_id", flat=True))
|
profile_post_ids = list(profile.posts.values_list("post_id", flat=True))
|
||||||
|
|
@ -127,6 +127,27 @@ class NotificationsView(APIView):
|
||||||
)
|
)
|
||||||
counts["replies"] = len(replies)
|
counts["replies"] = len(replies)
|
||||||
|
|
||||||
|
# 4. Get boosts (if not filtering or filtering for boosts)
|
||||||
|
# Build include patterns (feed#post_id) for all posts from this profile
|
||||||
|
include_patterns = [f"{feed_url}#{post_id}" for post_id in profile_post_ids]
|
||||||
|
|
||||||
|
if not notification_type or notification_type == "boost":
|
||||||
|
boosts = (
|
||||||
|
Post.objects.filter(include__in=include_patterns)
|
||||||
|
.select_related("profile")
|
||||||
|
.order_by("-post_id")
|
||||||
|
)
|
||||||
|
for boost in boosts:
|
||||||
|
notifications_data.append(
|
||||||
|
{
|
||||||
|
"type": "boost",
|
||||||
|
"post": f"{boost.profile.feed}#{boost.post_id}",
|
||||||
|
"boosted": boost.include,
|
||||||
|
"_sort_key": boost.post_id,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
counts["boosts"] = len(boosts)
|
||||||
|
|
||||||
# Sort all notifications by post_id (most recent first)
|
# Sort all notifications by post_id (most recent first)
|
||||||
notifications_data.sort(key=lambda x: x["_sort_key"], reverse=True)
|
notifications_data.sort(key=lambda x: x["_sort_key"], reverse=True)
|
||||||
|
|
||||||
|
|
|
||||||
0
app/profile/__init__.py
Normal file
0
app/profile/__init__.py
Normal file
6
app/profile/apps.py
Normal file
6
app/profile/apps.py
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
from django.apps import AppConfig
|
||||||
|
|
||||||
|
|
||||||
|
class ProfileConfig(AppConfig):
|
||||||
|
default_auto_field = "django.db.models.BigAutoField"
|
||||||
|
name = "app.profile"
|
||||||
176
app/profile/tests.py
Normal file
176
app/profile/tests.py
Normal file
|
|
@ -0,0 +1,176 @@
|
||||||
|
from django.test import TestCase
|
||||||
|
from rest_framework.test import APIClient
|
||||||
|
from rest_framework import status
|
||||||
|
|
||||||
|
from app.feeds.models import Follow, Profile
|
||||||
|
|
||||||
|
|
||||||
|
class ProfileViewTest(TestCase):
|
||||||
|
"""Test cases for the ProfileView API using Given/When/Then structure."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.client = APIClient()
|
||||||
|
self.url = "/profile/"
|
||||||
|
|
||||||
|
self.profile_main = Profile.objects.create(
|
||||||
|
feed="https://example.com/social.org",
|
||||||
|
title="Example Profile",
|
||||||
|
nick="example_user",
|
||||||
|
)
|
||||||
|
self.profile_alice = Profile.objects.create(
|
||||||
|
feed="https://alice.org/social.org",
|
||||||
|
title="Alice",
|
||||||
|
nick="alice",
|
||||||
|
)
|
||||||
|
self.profile_bob = Profile.objects.create(
|
||||||
|
feed="https://bob.org/social.org",
|
||||||
|
title="Bob",
|
||||||
|
nick="bob",
|
||||||
|
)
|
||||||
|
self.profile_carol = Profile.objects.create(
|
||||||
|
feed="https://carol.org/social.org",
|
||||||
|
title="Carol",
|
||||||
|
nick="carol",
|
||||||
|
)
|
||||||
|
|
||||||
|
Follow.objects.create(follower=self.profile_alice, followed=self.profile_main)
|
||||||
|
Follow.objects.create(follower=self.profile_bob, followed=self.profile_main)
|
||||||
|
|
||||||
|
def test_get_profile_with_followers(self):
|
||||||
|
"""Test GET /profile/?feed=<url> returns correct followers list."""
|
||||||
|
# Given: A profile followed by two users
|
||||||
|
feed_url = self.profile_main.feed
|
||||||
|
|
||||||
|
# When: We request the profile
|
||||||
|
response = self.client.get(self.url, {"feed": feed_url})
|
||||||
|
|
||||||
|
# Then: Response is successful
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
self.assertEqual(response.data["type"], "Success")
|
||||||
|
self.assertEqual(response.data["errors"], [])
|
||||||
|
|
||||||
|
# Then: Data contains feed and followers
|
||||||
|
data = response.data["data"]
|
||||||
|
self.assertEqual(data["feed"], feed_url)
|
||||||
|
self.assertIsInstance(data["followers"], list)
|
||||||
|
self.assertIn(self.profile_alice.feed, data["followers"])
|
||||||
|
self.assertIn(self.profile_bob.feed, data["followers"])
|
||||||
|
self.assertNotIn(self.profile_carol.feed, data["followers"])
|
||||||
|
|
||||||
|
# Then: Meta contains correct counts
|
||||||
|
meta = response.data["meta"]
|
||||||
|
self.assertEqual(meta["feed"], feed_url)
|
||||||
|
self.assertEqual(meta["total_followers"], 2)
|
||||||
|
|
||||||
|
def test_get_profile_no_followers(self):
|
||||||
|
"""Test GET /profile/ returns empty followers for profile with no followers."""
|
||||||
|
# Given: A profile with no followers
|
||||||
|
feed_url = self.profile_carol.feed
|
||||||
|
|
||||||
|
# When: We request the profile
|
||||||
|
response = self.client.get(self.url, {"feed": feed_url})
|
||||||
|
|
||||||
|
# Then: Response is successful with empty followers
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
self.assertEqual(response.data["type"], "Success")
|
||||||
|
self.assertEqual(response.data["data"]["followers"], [])
|
||||||
|
self.assertEqual(response.data["meta"]["total_followers"], 0)
|
||||||
|
|
||||||
|
def test_get_profile_not_found(self):
|
||||||
|
"""Test GET /profile/ returns 404 for non-existent feed."""
|
||||||
|
# Given: A feed URL not registered in the relay
|
||||||
|
nonexistent_feed = "https://nonexistent.com/social.org"
|
||||||
|
|
||||||
|
# When: We request the profile
|
||||||
|
response = self.client.get(self.url, {"feed": nonexistent_feed})
|
||||||
|
|
||||||
|
# Then: We get 404
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn("Profile not found", response.data["errors"][0])
|
||||||
|
self.assertIsNone(response.data["data"])
|
||||||
|
|
||||||
|
def test_get_profile_missing_feed_param(self):
|
||||||
|
"""Test GET /profile/ returns 400 when feed param is missing."""
|
||||||
|
# When: We request without the feed parameter
|
||||||
|
response = self.client.get(self.url)
|
||||||
|
|
||||||
|
# Then: We get 400
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
|
||||||
|
self.assertEqual(response.data["type"], "Error")
|
||||||
|
self.assertIn("required", response.data["errors"][0])
|
||||||
|
|
||||||
|
def test_get_profile_response_format(self):
|
||||||
|
"""Test GET /profile/ response matches README specification."""
|
||||||
|
# Given: A registered profile
|
||||||
|
feed_url = self.profile_main.feed
|
||||||
|
|
||||||
|
# When: We request the profile
|
||||||
|
response = self.client.get(self.url, {"feed": feed_url})
|
||||||
|
|
||||||
|
# Then: Response keys are correct
|
||||||
|
self.assertIn("type", response.data)
|
||||||
|
self.assertIn("errors", response.data)
|
||||||
|
self.assertIn("data", response.data)
|
||||||
|
self.assertIn("meta", response.data)
|
||||||
|
self.assertIn("_links", response.data)
|
||||||
|
|
||||||
|
data = response.data["data"]
|
||||||
|
self.assertIn("feed", data)
|
||||||
|
self.assertIn("followers", data)
|
||||||
|
|
||||||
|
meta = response.data["meta"]
|
||||||
|
self.assertIn("feed", meta)
|
||||||
|
self.assertIn("total_followers", meta)
|
||||||
|
|
||||||
|
links = response.data["_links"]
|
||||||
|
self.assertIn("self", links)
|
||||||
|
self.assertIn("href", links["self"])
|
||||||
|
self.assertIn("method", links["self"])
|
||||||
|
self.assertEqual(links["self"]["method"], "GET")
|
||||||
|
|
||||||
|
def test_get_profile_self_link_encoded(self):
|
||||||
|
"""Test that the self link URL-encodes the feed parameter."""
|
||||||
|
# Given: A feed URL with special characters
|
||||||
|
feed_url = self.profile_main.feed
|
||||||
|
|
||||||
|
# When: We request the profile
|
||||||
|
response = self.client.get(self.url, {"feed": feed_url})
|
||||||
|
|
||||||
|
# Then: Self link is URL-encoded
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
href = response.data["_links"]["self"]["href"]
|
||||||
|
self.assertIn("%3A", href) # : is encoded
|
||||||
|
self.assertIn("%2F", href) # / is encoded
|
||||||
|
|
||||||
|
def test_get_profile_http_methods(self):
|
||||||
|
"""Test that only GET is allowed on the profile endpoint."""
|
||||||
|
# Given: A valid feed URL
|
||||||
|
params = {"feed": self.profile_main.feed}
|
||||||
|
|
||||||
|
# When/Then: Non-GET methods return 405
|
||||||
|
self.assertEqual(
|
||||||
|
self.client.post(self.url, params).status_code,
|
||||||
|
status.HTTP_405_METHOD_NOT_ALLOWED,
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
self.client.put(self.url, params).status_code,
|
||||||
|
status.HTTP_405_METHOD_NOT_ALLOWED,
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
self.client.delete(self.url, params).status_code,
|
||||||
|
status.HTTP_405_METHOD_NOT_ALLOWED,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_get_profile_followers_are_feeds_only(self):
|
||||||
|
"""Test that followers list contains only feed URLs, not profile metadata."""
|
||||||
|
# Given: A profile with a follower
|
||||||
|
feed_url = self.profile_main.feed
|
||||||
|
|
||||||
|
# When: We request the profile
|
||||||
|
response = self.client.get(self.url, {"feed": feed_url})
|
||||||
|
|
||||||
|
# Then: Each follower entry is a plain URL string
|
||||||
|
for follower in response.data["data"]["followers"]:
|
||||||
|
self.assertIsInstance(follower, str)
|
||||||
|
self.assertTrue(follower.startswith("http"))
|
||||||
7
app/profile/urls.py
Normal file
7
app/profile/urls.py
Normal file
|
|
@ -0,0 +1,7 @@
|
||||||
|
from django.urls import path
|
||||||
|
|
||||||
|
from app.profile.views import ProfileView
|
||||||
|
|
||||||
|
urlpatterns = [
|
||||||
|
path("", ProfileView.as_view(), name="profile"),
|
||||||
|
]
|
||||||
76
app/profile/views.py
Normal file
76
app/profile/views.py
Normal file
|
|
@ -0,0 +1,76 @@
|
||||||
|
from urllib.parse import quote
|
||||||
|
|
||||||
|
from rest_framework import status
|
||||||
|
from rest_framework.response import Response
|
||||||
|
from rest_framework.views import APIView
|
||||||
|
from django.core.cache import cache
|
||||||
|
|
||||||
|
from app.feeds.models import Follow, Profile
|
||||||
|
|
||||||
|
|
||||||
|
class ProfileView(APIView):
|
||||||
|
"""Get profile information including followers for a given feed URL."""
|
||||||
|
|
||||||
|
def get(self, request):
|
||||||
|
feed_url = request.query_params.get("feed")
|
||||||
|
|
||||||
|
if not feed_url or not feed_url.strip():
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Error",
|
||||||
|
"errors": ["Feed URL parameter is required"],
|
||||||
|
"data": None,
|
||||||
|
},
|
||||||
|
status=status.HTTP_400_BAD_REQUEST,
|
||||||
|
)
|
||||||
|
|
||||||
|
feed_url = feed_url.strip()
|
||||||
|
|
||||||
|
cache_key = f"profile_{feed_url}"
|
||||||
|
cached_response = cache.get(cache_key)
|
||||||
|
if cached_response is not None:
|
||||||
|
return Response(cached_response, status=status.HTTP_200_OK)
|
||||||
|
|
||||||
|
try:
|
||||||
|
profile = Profile.objects.get(feed=feed_url)
|
||||||
|
except Profile.DoesNotExist:
|
||||||
|
return Response(
|
||||||
|
{
|
||||||
|
"type": "Error",
|
||||||
|
"errors": ["Profile not found for the given feed URL"],
|
||||||
|
"data": None,
|
||||||
|
},
|
||||||
|
status=status.HTTP_404_NOT_FOUND,
|
||||||
|
)
|
||||||
|
|
||||||
|
followers = (
|
||||||
|
Follow.objects.filter(followed=profile)
|
||||||
|
.select_related("follower")
|
||||||
|
.order_by("follower__feed")
|
||||||
|
)
|
||||||
|
follower_feeds = [f.follower.feed for f in followers]
|
||||||
|
|
||||||
|
encoded_feed_url = quote(feed_url, safe="")
|
||||||
|
|
||||||
|
response_data = {
|
||||||
|
"type": "Success",
|
||||||
|
"errors": [],
|
||||||
|
"data": {
|
||||||
|
"feed": feed_url,
|
||||||
|
"followers": follower_feeds,
|
||||||
|
},
|
||||||
|
"meta": {
|
||||||
|
"feed": feed_url,
|
||||||
|
"total_followers": len(follower_feeds),
|
||||||
|
},
|
||||||
|
"_links": {
|
||||||
|
"self": {
|
||||||
|
"href": f"/profile/?feed={encoded_feed_url}",
|
||||||
|
"method": "GET",
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
cache.set(cache_key, response_data, None)
|
||||||
|
|
||||||
|
return Response(response_data, status=status.HTTP_200_OK)
|
||||||
|
|
@ -15,6 +15,11 @@ def root_view(request):
|
||||||
"self": {"href": "/", "method": "GET"},
|
"self": {"href": "/", "method": "GET"},
|
||||||
"feeds": {"href": "/feeds/", "method": "GET"},
|
"feeds": {"href": "/feeds/", "method": "GET"},
|
||||||
"add-feed": {"href": "/feeds/", "method": "POST"},
|
"add-feed": {"href": "/feeds/", "method": "POST"},
|
||||||
|
"feed-content": {
|
||||||
|
"href": "/feed-content/?feed={feed_url}",
|
||||||
|
"method": "GET",
|
||||||
|
"templated": True,
|
||||||
|
},
|
||||||
"mentions": {
|
"mentions": {
|
||||||
"href": "/mentions/?feed={feed_url}",
|
"href": "/mentions/?feed={feed_url}",
|
||||||
"method": "GET",
|
"method": "GET",
|
||||||
|
|
@ -30,6 +35,15 @@ def root_view(request):
|
||||||
"method": "GET",
|
"method": "GET",
|
||||||
"templated": True,
|
"templated": True,
|
||||||
},
|
},
|
||||||
|
"sse-notifications": {
|
||||||
|
"href": "/sse/notifications/?feed={feed_url}",
|
||||||
|
"method": "GET",
|
||||||
|
"templated": True,
|
||||||
|
},
|
||||||
|
"sse-notifications-all": {
|
||||||
|
"href": "/sse/notifications/",
|
||||||
|
"method": "GET",
|
||||||
|
},
|
||||||
"reactions": {
|
"reactions": {
|
||||||
"href": "/reactions/?feed={feed_url}",
|
"href": "/reactions/?feed={feed_url}",
|
||||||
"method": "GET",
|
"method": "GET",
|
||||||
|
|
@ -40,6 +54,16 @@ def root_view(request):
|
||||||
"method": "GET",
|
"method": "GET",
|
||||||
"templated": True,
|
"templated": True,
|
||||||
},
|
},
|
||||||
|
"boosts": {
|
||||||
|
"href": "/boosts/?post={post_url}",
|
||||||
|
"method": "GET",
|
||||||
|
"templated": True,
|
||||||
|
},
|
||||||
|
"interactions": {
|
||||||
|
"href": "/interactions/?post={post_url}",
|
||||||
|
"method": "GET",
|
||||||
|
"templated": True,
|
||||||
|
},
|
||||||
"search": {
|
"search": {
|
||||||
"href": "/search/?q={query}",
|
"href": "/search/?q={query}",
|
||||||
"method": "GET",
|
"method": "GET",
|
||||||
|
|
@ -62,6 +86,30 @@ def root_view(request):
|
||||||
"method": "GET",
|
"method": "GET",
|
||||||
"templated": True,
|
"templated": True,
|
||||||
},
|
},
|
||||||
|
"profile": {
|
||||||
|
"href": "/profile/?feed={feed_url}",
|
||||||
|
"method": "GET",
|
||||||
|
"templated": True,
|
||||||
|
},
|
||||||
|
"rss": {
|
||||||
|
"href": "/rss.xml",
|
||||||
|
"method": "GET",
|
||||||
|
"description": "RSS feed of latest posts (supports ?tag={tag} and ?feed={feed_url} filters)",
|
||||||
|
},
|
||||||
|
"stats": {"href": "/stats/", "method": "GET"},
|
||||||
|
"bridge": {"href": "/bridge/", "method": "GET"},
|
||||||
|
"bridge-activitypub": {
|
||||||
|
"href": "/bridge/activitypub/@{user}@{instance}/",
|
||||||
|
"method": "GET",
|
||||||
|
"templated": True,
|
||||||
|
"description": "ActivityPub account as a virtual social.org feed",
|
||||||
|
},
|
||||||
|
"bridge-rss": {
|
||||||
|
"href": "/bridge/rss/?url={feed_url}",
|
||||||
|
"method": "GET",
|
||||||
|
"templated": True,
|
||||||
|
"description": "RSS/Atom feed as a virtual social.org feed",
|
||||||
|
},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,8 @@ class UTF8JSONRenderer(JSONRenderer):
|
||||||
"""
|
"""
|
||||||
JSON renderer that preserves UTF-8 characters (like emojis) instead of escaping them.
|
JSON renderer that preserves UTF-8 characters (like emojis) instead of escaping them.
|
||||||
"""
|
"""
|
||||||
charset = 'utf-8'
|
|
||||||
|
charset = "utf-8"
|
||||||
|
|
||||||
def render(self, data, accepted_media_type=None, renderer_context=None):
|
def render(self, data, accepted_media_type=None, renderer_context=None):
|
||||||
if data is None:
|
if data is None:
|
||||||
|
|
@ -16,16 +17,11 @@ class UTF8JSONRenderer(JSONRenderer):
|
||||||
indent = self.get_indent(accepted_media_type, renderer_context)
|
indent = self.get_indent(accepted_media_type, renderer_context)
|
||||||
|
|
||||||
if indent is None:
|
if indent is None:
|
||||||
separators = (',', ':')
|
separators = (",", ":")
|
||||||
else:
|
else:
|
||||||
separators = (',', ': ')
|
separators = (",", ": ")
|
||||||
|
|
||||||
ret = json.dumps(
|
ret = json.dumps(data, ensure_ascii=False, indent=indent, separators=separators)
|
||||||
data,
|
|
||||||
ensure_ascii=False,
|
|
||||||
indent=indent,
|
|
||||||
separators=separators
|
|
||||||
)
|
|
||||||
|
|
||||||
# Handle invalid surrogate characters by using 'surrogatepass' error handler
|
# Handle invalid surrogate characters by using 'surrogatepass' error handler
|
||||||
return ret.encode('utf-8', errors='surrogatepass')
|
return ret.encode("utf-8", errors="surrogatepass")
|
||||||
|
|
|
||||||
24
app/reactions/utils.py
Normal file
24
app/reactions/utils.py
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
"""
|
||||||
|
Utility functions for reactions
|
||||||
|
"""
|
||||||
|
|
||||||
|
from app.feeds.models import Post
|
||||||
|
|
||||||
|
|
||||||
|
def get_reactions_for_post(post_url: str):
|
||||||
|
"""
|
||||||
|
Get all reactions for a specific post.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
post_url: The post URL in format feed_url#post_id
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
QuerySet of Post objects that are reactions to the given post
|
||||||
|
"""
|
||||||
|
return (
|
||||||
|
Post.objects.filter(reply_to=post_url, mood__isnull=False)
|
||||||
|
.exclude(mood="")
|
||||||
|
.exclude(poll_votes__isnull=False)
|
||||||
|
.select_related("profile")
|
||||||
|
.order_by("-post_id")
|
||||||
|
)
|
||||||
|
|
@ -12,6 +12,7 @@ logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
class ReactionsView(APIView):
|
class ReactionsView(APIView):
|
||||||
"""Get reactions for a specific feed URL"""
|
"""Get reactions for a specific feed URL"""
|
||||||
|
|
||||||
renderer_classes = [UTF8JSONRenderer]
|
renderer_classes = [UTF8JSONRenderer]
|
||||||
|
|
||||||
def get(self, request):
|
def get(self, request):
|
||||||
|
|
|
||||||
26
app/replies/utils.py
Normal file
26
app/replies/utils.py
Normal file
|
|
@ -0,0 +1,26 @@
|
||||||
|
"""
|
||||||
|
Utility functions for replies
|
||||||
|
"""
|
||||||
|
|
||||||
|
from django.db.models import Q
|
||||||
|
from app.feeds.models import Post
|
||||||
|
|
||||||
|
|
||||||
|
def get_direct_replies_for_post(post_url: str):
|
||||||
|
"""
|
||||||
|
Get direct replies for a specific post.
|
||||||
|
Excludes reactions (posts with mood) and poll votes.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
post_url: The post URL in format feed_url#post_id
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
QuerySet of Post objects that are direct replies to the given post
|
||||||
|
"""
|
||||||
|
return (
|
||||||
|
Post.objects.filter(reply_to=post_url)
|
||||||
|
.filter(Q(mood="") | Q(mood__isnull=True))
|
||||||
|
.exclude(poll_votes__isnull=False)
|
||||||
|
.select_related("profile")
|
||||||
|
.order_by("-post_id")
|
||||||
|
)
|
||||||
0
app/rss/__init__.py
Normal file
0
app/rss/__init__.py
Normal file
6
app/rss/apps.py
Normal file
6
app/rss/apps.py
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
from django.apps import AppConfig
|
||||||
|
|
||||||
|
|
||||||
|
class RssConfig(AppConfig):
|
||||||
|
default_auto_field = "django.db.models.BigAutoField"
|
||||||
|
name = "app.rss"
|
||||||
541
app/rss/tests.py
Normal file
541
app/rss/tests.py
Normal file
|
|
@ -0,0 +1,541 @@
|
||||||
|
from django.test import TestCase, override_settings
|
||||||
|
from rest_framework.test import APIClient
|
||||||
|
from rest_framework import status
|
||||||
|
from xml.etree import ElementTree as ET
|
||||||
|
|
||||||
|
from app.feeds.models import Profile, Post
|
||||||
|
|
||||||
|
|
||||||
|
class RSSFeedTest(TestCase):
|
||||||
|
"""Test cases for the RSS Feed using Given/When/Then structure."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.client = APIClient()
|
||||||
|
self.rss_url = "/rss.xml"
|
||||||
|
|
||||||
|
# Create test profiles
|
||||||
|
self.profile1 = Profile.objects.create(
|
||||||
|
feed="https://example.com/social.org",
|
||||||
|
title="Example Profile",
|
||||||
|
nick="example_user",
|
||||||
|
description="Test profile 1",
|
||||||
|
)
|
||||||
|
self.profile2 = Profile.objects.create(
|
||||||
|
feed="https://test.com/social.org",
|
||||||
|
title="Test Profile",
|
||||||
|
nick="test_user",
|
||||||
|
description="Test profile 2",
|
||||||
|
)
|
||||||
|
self.profile3 = Profile.objects.create(
|
||||||
|
feed="https://third.com/social.org",
|
||||||
|
title="Third Profile",
|
||||||
|
nick="third_user",
|
||||||
|
description="Test profile 3",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create test posts with different content and tags
|
||||||
|
self.post1 = Post.objects.create(
|
||||||
|
profile=self.profile1,
|
||||||
|
post_id="2025-01-01T12:00:00+00:00",
|
||||||
|
content="This post is about Emacs and org-mode",
|
||||||
|
tags="emacs org-mode",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.post2 = Post.objects.create(
|
||||||
|
profile=self.profile2,
|
||||||
|
post_id="2025-01-01T13:00:00+00:00",
|
||||||
|
content="Learning Python programming language",
|
||||||
|
tags="python programming",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.post3 = Post.objects.create(
|
||||||
|
profile=self.profile3,
|
||||||
|
post_id="2025-01-01T14:00:00+00:00",
|
||||||
|
content="Django web framework with Python",
|
||||||
|
tags="django python web",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.post4 = Post.objects.create(
|
||||||
|
profile=self.profile1,
|
||||||
|
post_id="2025-01-01T15:00:00+00:00",
|
||||||
|
content="Emacs configuration and setup",
|
||||||
|
tags="emacs configuration",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.post5 = Post.objects.create(
|
||||||
|
profile=self.profile2,
|
||||||
|
post_id="2025-01-01T16:00:00+00:00",
|
||||||
|
content="JavaScript and React development",
|
||||||
|
tags="javascript react frontend",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_rss_feed_all_posts_success(self):
|
||||||
|
"""Test GET /rss.xml returns RSS feed with all posts."""
|
||||||
|
# Given: Posts exist in the database
|
||||||
|
|
||||||
|
# When: We request the RSS feed
|
||||||
|
response = self.client.get(self.rss_url)
|
||||||
|
|
||||||
|
# Then: Should return 200 and valid RSS
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
self.assertEqual(response["Content-Type"], "application/rss+xml; charset=utf-8")
|
||||||
|
|
||||||
|
# Parse XML and verify structure
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
self.assertEqual(root.tag, "rss")
|
||||||
|
self.assertEqual(root.get("version"), "2.0")
|
||||||
|
|
||||||
|
# Verify channel
|
||||||
|
channel = root.find("channel")
|
||||||
|
self.assertIsNotNone(channel)
|
||||||
|
|
||||||
|
# Verify title
|
||||||
|
title = channel.find("title")
|
||||||
|
self.assertIsNotNone(title)
|
||||||
|
self.assertEqual(title.text, "Org Social Relay - Latest Posts")
|
||||||
|
|
||||||
|
# Verify items
|
||||||
|
items = channel.findall("item")
|
||||||
|
self.assertEqual(len(items), 5) # All 5 posts
|
||||||
|
|
||||||
|
# Verify first item (most recent)
|
||||||
|
first_item = items[0]
|
||||||
|
guid = first_item.find("guid")
|
||||||
|
self.assertIsNotNone(guid)
|
||||||
|
self.assertIn(self.post5.post_id, guid.text)
|
||||||
|
|
||||||
|
def test_rss_feed_filtered_by_tag(self):
|
||||||
|
"""Test GET /rss.xml?tag=<tag> returns RSS feed filtered by tag."""
|
||||||
|
# Given: Posts with various tags
|
||||||
|
|
||||||
|
# When: We request the RSS feed filtered by tag "emacs"
|
||||||
|
response = self.client.get(self.rss_url, {"tag": "emacs"})
|
||||||
|
|
||||||
|
# Then: Should return 200 and filtered RSS
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
|
||||||
|
# Parse XML
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
channel = root.find("channel")
|
||||||
|
|
||||||
|
# Verify title includes tag
|
||||||
|
title = channel.find("title")
|
||||||
|
self.assertIn("emacs", title.text.lower())
|
||||||
|
|
||||||
|
# Verify only posts with "emacs" tag are included
|
||||||
|
items = channel.findall("item")
|
||||||
|
self.assertEqual(len(items), 2) # post1 and post4
|
||||||
|
|
||||||
|
# Verify all items have the emacs tag
|
||||||
|
for item in items:
|
||||||
|
categories = [cat.text for cat in item.findall("category")]
|
||||||
|
self.assertIn("emacs", categories)
|
||||||
|
|
||||||
|
def test_rss_feed_filtered_by_feed(self):
|
||||||
|
"""Test GET /rss.xml?feed=<feed_url> returns RSS feed filtered by author feed."""
|
||||||
|
# Given: Posts from different profiles
|
||||||
|
|
||||||
|
# When: We request the RSS feed filtered by profile1's feed
|
||||||
|
response = self.client.get(self.rss_url, {"feed": self.profile1.feed})
|
||||||
|
|
||||||
|
# Then: Should return 200 and filtered RSS
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
|
||||||
|
# Parse XML
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
channel = root.find("channel")
|
||||||
|
|
||||||
|
# Verify title includes feed
|
||||||
|
title = channel.find("title")
|
||||||
|
self.assertIn(self.profile1.feed, title.text)
|
||||||
|
|
||||||
|
# Verify only posts from profile1 are included
|
||||||
|
items = channel.findall("item")
|
||||||
|
self.assertEqual(len(items), 2) # post1 and post4
|
||||||
|
|
||||||
|
# Verify all items are from profile1
|
||||||
|
for item in items:
|
||||||
|
link = item.find("link")
|
||||||
|
self.assertIn(self.profile1.feed, link.text)
|
||||||
|
|
||||||
|
def test_rss_feed_item_structure(self):
|
||||||
|
"""Test that RSS feed items have correct structure."""
|
||||||
|
# Given: Posts exist
|
||||||
|
|
||||||
|
# When: We request the RSS feed
|
||||||
|
response = self.client.get(self.rss_url)
|
||||||
|
|
||||||
|
# Parse XML
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
channel = root.find("channel")
|
||||||
|
items = channel.findall("item")
|
||||||
|
|
||||||
|
# Then: Each item should have required fields
|
||||||
|
for item in items:
|
||||||
|
# Required fields
|
||||||
|
self.assertIsNotNone(item.find("title"))
|
||||||
|
self.assertIsNotNone(item.find("link"))
|
||||||
|
self.assertIsNotNone(item.find("description"))
|
||||||
|
self.assertIsNotNone(item.find("pubDate"))
|
||||||
|
self.assertIsNotNone(item.find("guid"))
|
||||||
|
self.assertIsNotNone(item.find("author"))
|
||||||
|
|
||||||
|
def test_rss_feed_limit_200_posts(self):
|
||||||
|
"""Test that RSS feed is limited to 200 posts as per specification."""
|
||||||
|
# Given: More than 200 posts
|
||||||
|
for i in range(201):
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile1,
|
||||||
|
post_id=f"2025-01-02T{i // 60:02d}:{i % 60:02d}:00+00:00",
|
||||||
|
content=f"Post number {i}",
|
||||||
|
tags="test",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: We request the RSS feed
|
||||||
|
response = self.client.get(self.rss_url)
|
||||||
|
|
||||||
|
# Parse XML
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
channel = root.find("channel")
|
||||||
|
items = channel.findall("item")
|
||||||
|
|
||||||
|
# Then: Should be limited to 200 posts
|
||||||
|
self.assertEqual(len(items), 200)
|
||||||
|
|
||||||
|
def test_rss_feed_ordered_by_most_recent(self):
|
||||||
|
"""Test that RSS feed items are ordered from most recent to oldest."""
|
||||||
|
# Given: Posts with different timestamps
|
||||||
|
|
||||||
|
# When: We request the RSS feed
|
||||||
|
response = self.client.get(self.rss_url)
|
||||||
|
|
||||||
|
# Parse XML
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
channel = root.find("channel")
|
||||||
|
items = channel.findall("item")
|
||||||
|
|
||||||
|
# Then: First item should be the most recent post
|
||||||
|
first_item = items[0]
|
||||||
|
first_guid = first_item.find("guid").text
|
||||||
|
self.assertIn(self.post5.post_id, first_guid) # Most recent
|
||||||
|
|
||||||
|
# Last item should be the oldest post
|
||||||
|
last_item = items[-1]
|
||||||
|
last_guid = last_item.find("guid").text
|
||||||
|
self.assertIn(self.post1.post_id, last_guid) # Oldest
|
||||||
|
|
||||||
|
def test_rss_feed_categories_from_tags(self):
|
||||||
|
"""Test that post tags are converted to RSS categories."""
|
||||||
|
# Given: Post with multiple tags
|
||||||
|
|
||||||
|
# When: We request the RSS feed
|
||||||
|
response = self.client.get(self.rss_url)
|
||||||
|
|
||||||
|
# Parse XML
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
channel = root.find("channel")
|
||||||
|
items = channel.findall("item")
|
||||||
|
|
||||||
|
# Find item for post3 (has multiple tags)
|
||||||
|
post3_item = None
|
||||||
|
for item in items:
|
||||||
|
guid = item.find("guid").text
|
||||||
|
if self.post3.post_id in guid:
|
||||||
|
post3_item = item
|
||||||
|
break
|
||||||
|
|
||||||
|
self.assertIsNotNone(post3_item)
|
||||||
|
|
||||||
|
# Then: Categories should match tags
|
||||||
|
categories = [cat.text for cat in post3_item.findall("category")]
|
||||||
|
expected_tags = self.post3.tags.split()
|
||||||
|
self.assertEqual(set(categories), set(expected_tags))
|
||||||
|
|
||||||
|
def test_rss_feed_guid_is_post_url(self):
|
||||||
|
"""Test that GUID is the post URL."""
|
||||||
|
# Given: Posts exist
|
||||||
|
|
||||||
|
# When: We request the RSS feed
|
||||||
|
response = self.client.get(self.rss_url)
|
||||||
|
|
||||||
|
# Parse XML
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
channel = root.find("channel")
|
||||||
|
items = channel.findall("item")
|
||||||
|
|
||||||
|
# Then: Each GUID should be feed#post_id format
|
||||||
|
for item in items:
|
||||||
|
guid = item.find("guid")
|
||||||
|
self.assertIsNotNone(guid)
|
||||||
|
self.assertIn("#", guid.text)
|
||||||
|
self.assertTrue(guid.get("isPermaLink") in ["true", None])
|
||||||
|
|
||||||
|
def test_rss_feed_content_type(self):
|
||||||
|
"""Test that RSS feed returns correct content type."""
|
||||||
|
# Given: RSS endpoint
|
||||||
|
|
||||||
|
# When: We request the RSS feed
|
||||||
|
response = self.client.get(self.rss_url)
|
||||||
|
|
||||||
|
# Then: Content-Type should be application/rss+xml
|
||||||
|
self.assertEqual(response["Content-Type"], "application/rss+xml; charset=utf-8")
|
||||||
|
|
||||||
|
def test_rss_feed_valid_xml(self):
|
||||||
|
"""Test that RSS feed returns valid XML."""
|
||||||
|
# Given: RSS endpoint
|
||||||
|
|
||||||
|
# When: We request the RSS feed
|
||||||
|
response = self.client.get(self.rss_url)
|
||||||
|
|
||||||
|
# Then: Should be parseable as XML without errors
|
||||||
|
try:
|
||||||
|
ET.fromstring(response.content)
|
||||||
|
except ET.ParseError as e:
|
||||||
|
self.fail(f"RSS feed is not valid XML: {e}")
|
||||||
|
|
||||||
|
def test_rss_feed_description_is_post_content(self):
|
||||||
|
"""Test that description contains the post content (converted to HTML)."""
|
||||||
|
# Given: Posts with content
|
||||||
|
|
||||||
|
# When: We request the RSS feed
|
||||||
|
response = self.client.get(self.rss_url)
|
||||||
|
|
||||||
|
# Parse XML
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
channel = root.find("channel")
|
||||||
|
items = channel.findall("item")
|
||||||
|
|
||||||
|
# Find item for post1
|
||||||
|
post1_item = None
|
||||||
|
for item in items:
|
||||||
|
guid = item.find("guid").text
|
||||||
|
if self.post1.post_id in guid:
|
||||||
|
post1_item = item
|
||||||
|
break
|
||||||
|
|
||||||
|
self.assertIsNotNone(post1_item)
|
||||||
|
|
||||||
|
# Then: Description should contain HTML-converted content
|
||||||
|
description = post1_item.find("description")
|
||||||
|
self.assertIsNotNone(description.text)
|
||||||
|
# Content is now converted to HTML, so we check for HTML elements
|
||||||
|
self.assertIn("<p>", description.text)
|
||||||
|
# The original content should be present in the HTML
|
||||||
|
self.assertIn("This post is about Emacs and org-mode", description.text)
|
||||||
|
|
||||||
|
def test_rss_feed_author_is_nick(self):
|
||||||
|
"""Test that author is the profile nick."""
|
||||||
|
# Given: Posts from different profiles
|
||||||
|
|
||||||
|
# When: We request the RSS feed
|
||||||
|
response = self.client.get(self.rss_url)
|
||||||
|
|
||||||
|
# Parse XML
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
channel = root.find("channel")
|
||||||
|
items = channel.findall("item")
|
||||||
|
|
||||||
|
# Find item for post1
|
||||||
|
post1_item = None
|
||||||
|
for item in items:
|
||||||
|
guid = item.find("guid").text
|
||||||
|
if self.post1.post_id in guid:
|
||||||
|
post1_item = item
|
||||||
|
break
|
||||||
|
|
||||||
|
self.assertIsNotNone(post1_item)
|
||||||
|
|
||||||
|
# Then: Author should be the profile nick
|
||||||
|
author = post1_item.find("author")
|
||||||
|
self.assertIn(self.profile1.nick, author.text)
|
||||||
|
|
||||||
|
def test_rss_feed_empty_when_no_posts(self):
|
||||||
|
"""Test RSS feed when there are no posts."""
|
||||||
|
# Given: No posts exist
|
||||||
|
Post.objects.all().delete()
|
||||||
|
|
||||||
|
# When: We request the RSS feed
|
||||||
|
response = self.client.get(self.rss_url)
|
||||||
|
|
||||||
|
# Then: Should still return valid RSS with no items
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
|
||||||
|
# Parse XML
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
channel = root.find("channel")
|
||||||
|
items = channel.findall("item")
|
||||||
|
|
||||||
|
# Should have 0 items
|
||||||
|
self.assertEqual(len(items), 0)
|
||||||
|
|
||||||
|
def test_rss_feed_link_is_post_url(self):
|
||||||
|
"""Test that link element is the post URL."""
|
||||||
|
# Given: Posts exist
|
||||||
|
|
||||||
|
# When: We request the RSS feed
|
||||||
|
response = self.client.get(self.rss_url)
|
||||||
|
|
||||||
|
# Parse XML
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
channel = root.find("channel")
|
||||||
|
items = channel.findall("item")
|
||||||
|
|
||||||
|
# Then: Each link should be feed#post_id format
|
||||||
|
for item in items:
|
||||||
|
link = item.find("link")
|
||||||
|
guid = item.find("guid")
|
||||||
|
# Link and GUID should be the same
|
||||||
|
self.assertEqual(link.text, guid.text)
|
||||||
|
self.assertIn("#", link.text)
|
||||||
|
|
||||||
|
def test_rss_feed_excludes_empty_content(self):
|
||||||
|
"""Test that RSS feed excludes posts with empty content (reactions, votes, etc.)."""
|
||||||
|
# Given: Posts with and without content
|
||||||
|
# Create posts with empty content (reactions, votes)
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile1,
|
||||||
|
post_id="2025-01-01T17:00:00+00:00",
|
||||||
|
content="", # Empty content (reaction/vote)
|
||||||
|
tags="reaction",
|
||||||
|
)
|
||||||
|
Post.objects.create(
|
||||||
|
profile=self.profile2,
|
||||||
|
post_id="2025-01-01T18:00:00+00:00",
|
||||||
|
content=" ", # Whitespace only
|
||||||
|
tags="vote",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: We request the RSS feed
|
||||||
|
response = self.client.get(self.rss_url)
|
||||||
|
|
||||||
|
# Parse XML
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
channel = root.find("channel")
|
||||||
|
items = channel.findall("item")
|
||||||
|
|
||||||
|
# Then: Should only include posts with content (5 original posts)
|
||||||
|
# Empty and whitespace-only posts should be excluded
|
||||||
|
self.assertEqual(len(items), 5)
|
||||||
|
|
||||||
|
# Verify all items have non-empty description
|
||||||
|
for item in items:
|
||||||
|
description = item.find("description")
|
||||||
|
# Description should exist and have content
|
||||||
|
self.assertIsNotNone(description.text)
|
||||||
|
# After HTML conversion, should have actual content
|
||||||
|
self.assertIn("<p>", description.text)
|
||||||
|
|
||||||
|
def test_rss_feed_excludes_bridge_posts(self):
|
||||||
|
"""Test that posts from bridge virtual feeds are excluded from the feed."""
|
||||||
|
# Given: A bridge profile with a post, alongside the regular posts
|
||||||
|
bridge_profile = Profile.objects.create(
|
||||||
|
feed=(
|
||||||
|
"https://relay.org-social.org/bridge/rss/"
|
||||||
|
"?url=https%3A%2F%2Fexample.org%2Ffeed.xml"
|
||||||
|
),
|
||||||
|
title="Bridged Feed",
|
||||||
|
nick="bridged_feed",
|
||||||
|
description="Bridge virtual feed",
|
||||||
|
)
|
||||||
|
bridge_post = Post.objects.create(
|
||||||
|
profile=bridge_profile,
|
||||||
|
post_id="2025-01-01T20:00:00+00:00",
|
||||||
|
content="Post coming from a bridged RSS feed",
|
||||||
|
tags="bridged",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: We request the global RSS feed
|
||||||
|
response = self.client.get(self.rss_url)
|
||||||
|
|
||||||
|
# Then: Only the posts from real profiles are included
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
items = root.find("channel").findall("item")
|
||||||
|
self.assertEqual(len(items), 5)
|
||||||
|
guids = [item.find("guid").text for item in items]
|
||||||
|
for guid in guids:
|
||||||
|
self.assertNotIn(bridge_post.post_id, guid)
|
||||||
|
|
||||||
|
def test_rss_feed_filtered_by_tag_excludes_bridge_posts(self):
|
||||||
|
"""Test that bridge posts are excluded from tag-filtered feeds too."""
|
||||||
|
# Given: A bridge profile with a post tagged like a regular post
|
||||||
|
bridge_profile = Profile.objects.create(
|
||||||
|
feed="https://relay.org-social.org/bridge/activitypub/@user@instance.tld/",
|
||||||
|
title="Bridged Account",
|
||||||
|
nick="bridged_account",
|
||||||
|
description="Bridge virtual feed",
|
||||||
|
)
|
||||||
|
Post.objects.create(
|
||||||
|
profile=bridge_profile,
|
||||||
|
post_id="2025-01-01T21:00:00+00:00",
|
||||||
|
content="Bridged post about Emacs",
|
||||||
|
tags="emacs",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: We request the RSS feed filtered by tag "emacs"
|
||||||
|
response = self.client.get(self.rss_url, {"tag": "emacs"})
|
||||||
|
|
||||||
|
# Then: Only the real posts with that tag are included
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
items = root.find("channel").findall("item")
|
||||||
|
self.assertEqual(len(items), 2) # post1 and post4
|
||||||
|
|
||||||
|
def test_rss_feed_explicit_bridge_feed_is_still_served(self):
|
||||||
|
"""Test that explicitly requesting a bridge feed still returns its posts."""
|
||||||
|
# Given: A bridge profile with a post
|
||||||
|
bridge_feed_url = (
|
||||||
|
"https://relay.org-social.org/bridge/rss/"
|
||||||
|
"?url=https%3A%2F%2Fexample.org%2Ffeed.xml"
|
||||||
|
)
|
||||||
|
bridge_profile = Profile.objects.create(
|
||||||
|
feed=bridge_feed_url,
|
||||||
|
title="Bridged Feed",
|
||||||
|
nick="bridged_feed",
|
||||||
|
description="Bridge virtual feed",
|
||||||
|
)
|
||||||
|
bridge_post = Post.objects.create(
|
||||||
|
profile=bridge_profile,
|
||||||
|
post_id="2025-01-01T20:00:00+00:00",
|
||||||
|
content="Post coming from a bridged RSS feed",
|
||||||
|
tags="bridged",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: We request the RSS feed filtered by the bridge feed itself
|
||||||
|
response = self.client.get(self.rss_url, {"feed": bridge_feed_url})
|
||||||
|
|
||||||
|
# Then: The bridge post is served
|
||||||
|
root = ET.fromstring(response.content)
|
||||||
|
items = root.find("channel").findall("item")
|
||||||
|
self.assertEqual(len(items), 1)
|
||||||
|
self.assertIn(bridge_post.post_id, items[0].find("guid").text)
|
||||||
|
|
||||||
|
@override_settings(
|
||||||
|
CACHES={
|
||||||
|
"default": {
|
||||||
|
"BACKEND": "django.core.cache.backends.locmem.LocMemCache",
|
||||||
|
"LOCATION": "test-cache",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
def test_rss_feed_xml_caching(self):
|
||||||
|
"""Test that RSS feed XML response is cached for performance."""
|
||||||
|
from django.core.cache import cache
|
||||||
|
|
||||||
|
# Given: Posts exist and cache is cleared
|
||||||
|
cache_key = "rss_xml_all"
|
||||||
|
cache.delete(cache_key)
|
||||||
|
|
||||||
|
# When: We request the RSS feed for the first time
|
||||||
|
response1 = self.client.get(self.rss_url)
|
||||||
|
self.assertEqual(response1.status_code, status.HTTP_200_OK)
|
||||||
|
|
||||||
|
# Then: The XML should be cached
|
||||||
|
cached_xml = cache.get(cache_key)
|
||||||
|
self.assertIsNotNone(cached_xml)
|
||||||
|
|
||||||
|
# When: We request the RSS feed again
|
||||||
|
response2 = self.client.get(self.rss_url)
|
||||||
|
self.assertEqual(response2.status_code, status.HTTP_200_OK)
|
||||||
|
|
||||||
|
# Then: Both responses should be identical
|
||||||
|
self.assertEqual(response1.content, response2.content)
|
||||||
|
self.assertEqual(response1["Content-Type"], response2["Content-Type"])
|
||||||
6
app/rss/urls.py
Normal file
6
app/rss/urls.py
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
from django.urls import path
|
||||||
|
from .views import LatestPostsFeed
|
||||||
|
|
||||||
|
urlpatterns = [
|
||||||
|
path("", LatestPostsFeed(), name="rss_feed"),
|
||||||
|
]
|
||||||
238
app/rss/views.py
Normal file
238
app/rss/views.py
Normal file
|
|
@ -0,0 +1,238 @@
|
||||||
|
from django.contrib.syndication.views import Feed
|
||||||
|
from django.utils.feedgenerator import Rss201rev2Feed
|
||||||
|
from django.core.cache import cache
|
||||||
|
from django.conf import settings
|
||||||
|
from django.http import HttpResponse
|
||||||
|
import hashlib
|
||||||
|
import re
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from app.bridge.models import bridge_urls_q
|
||||||
|
from app.feeds.models import Post
|
||||||
|
|
||||||
|
try:
|
||||||
|
from orgpython import to_html as org_to_html
|
||||||
|
|
||||||
|
HAS_ORGPYTHON = True
|
||||||
|
except ImportError:
|
||||||
|
HAS_ORGPYTHON = False
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class CustomRss201rev2Feed(Rss201rev2Feed):
|
||||||
|
"""Custom RSS generator that adds author field properly"""
|
||||||
|
|
||||||
|
def add_item_elements(self, handler, item):
|
||||||
|
"""Add item elements including author"""
|
||||||
|
super().add_item_elements(handler, item)
|
||||||
|
|
||||||
|
# Add author field if provided
|
||||||
|
if item.get("author"):
|
||||||
|
handler.addQuickElement("author", item["author"])
|
||||||
|
|
||||||
|
|
||||||
|
class LatestPostsFeed(Feed):
|
||||||
|
"""RSS feed for latest posts from Org Social Relay"""
|
||||||
|
|
||||||
|
feed_type = CustomRss201rev2Feed
|
||||||
|
|
||||||
|
def __call__(self, request, *args, **kwargs):
|
||||||
|
"""Override to add XML caching"""
|
||||||
|
# Build cache key for the XML response
|
||||||
|
tag = request.GET.get("tag", "")
|
||||||
|
feed_url = request.GET.get("feed", "")
|
||||||
|
|
||||||
|
cache_parts = ["rss_xml"]
|
||||||
|
if tag:
|
||||||
|
cache_parts.append(f"tag_{hashlib.md5(tag.encode()).hexdigest()[:8]}")
|
||||||
|
elif feed_url:
|
||||||
|
cache_parts.append(f"feed_{hashlib.md5(feed_url.encode()).hexdigest()[:8]}")
|
||||||
|
else:
|
||||||
|
cache_parts.append("all")
|
||||||
|
|
||||||
|
cache_key = "_".join(cache_parts)
|
||||||
|
|
||||||
|
# Try to get cached XML
|
||||||
|
cached_xml = cache.get(cache_key)
|
||||||
|
if cached_xml is not None:
|
||||||
|
return HttpResponse(
|
||||||
|
cached_xml, content_type="application/rss+xml; charset=utf-8"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Generate feed normally
|
||||||
|
response = super().__call__(request, *args, **kwargs)
|
||||||
|
|
||||||
|
# Cache the XML response for 5 minutes (300 seconds)
|
||||||
|
if response.status_code == 200:
|
||||||
|
cache.set(cache_key, response.content, 300)
|
||||||
|
|
||||||
|
return response
|
||||||
|
|
||||||
|
def get_object(self, request):
|
||||||
|
"""Process query parameters"""
|
||||||
|
tag = request.GET.get("tag")
|
||||||
|
feed_url = request.GET.get("feed")
|
||||||
|
return {"tag": tag, "feed": feed_url}
|
||||||
|
|
||||||
|
def title(self, obj):
|
||||||
|
"""Generate feed title based on filters"""
|
||||||
|
if obj["tag"]:
|
||||||
|
return f"Org Social Relay - Posts tagged with '{obj['tag']}'"
|
||||||
|
elif obj["feed"]:
|
||||||
|
return f"Org Social Relay - Posts from {obj['feed']}"
|
||||||
|
return "Org Social Relay - Latest Posts"
|
||||||
|
|
||||||
|
def link(self, obj):
|
||||||
|
"""Generate feed link"""
|
||||||
|
site_domain = settings.SITE_DOMAIN
|
||||||
|
protocol = "https" if not settings.DEBUG else "http"
|
||||||
|
base_url = f"{protocol}://{site_domain}/rss.xml"
|
||||||
|
|
||||||
|
if obj["tag"]:
|
||||||
|
return f"{base_url}?tag={obj['tag']}"
|
||||||
|
elif obj["feed"]:
|
||||||
|
return f"{base_url}?feed={obj['feed']}"
|
||||||
|
return base_url
|
||||||
|
|
||||||
|
def description(self, obj):
|
||||||
|
"""Generate feed description based on filters"""
|
||||||
|
if obj["tag"]:
|
||||||
|
return f"Latest posts from Org Social Relay tagged with '{obj['tag']}'"
|
||||||
|
elif obj["feed"]:
|
||||||
|
return f"Latest posts from {obj['feed']} on Org Social Relay"
|
||||||
|
return "Latest posts from all registered feeds on Org Social Relay"
|
||||||
|
|
||||||
|
def items(self, obj):
|
||||||
|
"""Return items for the feed, limited to 200 posts"""
|
||||||
|
# Build cache key based on filters
|
||||||
|
cache_parts = ["rss_feed"]
|
||||||
|
if obj["tag"]:
|
||||||
|
cache_parts.append(
|
||||||
|
f"tag_{hashlib.md5(obj['tag'].encode()).hexdigest()[:8]}"
|
||||||
|
)
|
||||||
|
elif obj["feed"]:
|
||||||
|
cache_parts.append(
|
||||||
|
f"feed_{hashlib.md5(obj['feed'].encode()).hexdigest()[:8]}"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
cache_parts.append("all")
|
||||||
|
|
||||||
|
cache_key = "_".join(cache_parts)
|
||||||
|
cached_posts = cache.get(cache_key)
|
||||||
|
|
||||||
|
if cached_posts is not None:
|
||||||
|
return cached_posts
|
||||||
|
|
||||||
|
# Build query - exclude posts with empty content (reactions, votes, etc.)
|
||||||
|
posts_query = (
|
||||||
|
Post.objects.select_related("profile")
|
||||||
|
.exclude(content__isnull=True)
|
||||||
|
.exclude(content__exact="")
|
||||||
|
.order_by("-created_at")
|
||||||
|
)
|
||||||
|
|
||||||
|
# Apply filters
|
||||||
|
if obj["tag"]:
|
||||||
|
# Search by specific tag (exact word match, case insensitive)
|
||||||
|
tag_escaped = re.escape(obj["tag"])
|
||||||
|
tag_pattern = rf"(^|[\s]){tag_escaped}([\s]|$)"
|
||||||
|
posts_query = posts_query.filter(tags__iregex=tag_pattern).exclude(
|
||||||
|
bridge_urls_q("profile__feed")
|
||||||
|
)
|
||||||
|
elif obj["feed"]:
|
||||||
|
# Filter by author feed. An explicitly requested bridge feed is
|
||||||
|
# still served: only aggregate feeds must leave bridges out.
|
||||||
|
posts_query = posts_query.filter(profile__feed=obj["feed"])
|
||||||
|
else:
|
||||||
|
posts_query = posts_query.exclude(bridge_urls_q("profile__feed"))
|
||||||
|
|
||||||
|
# Fetch more than 200 to account for filtering whitespace-only posts
|
||||||
|
posts_raw = list(posts_query[:250])
|
||||||
|
|
||||||
|
# Filter out posts with only whitespace content
|
||||||
|
posts = [p for p in posts_raw if p.content and p.content.strip()]
|
||||||
|
|
||||||
|
# Limit to 200 posts as per specification
|
||||||
|
posts = posts[:200]
|
||||||
|
|
||||||
|
# Cache permanently (will be cleared by scan_feeds task)
|
||||||
|
cache.set(cache_key, posts, None)
|
||||||
|
|
||||||
|
return posts
|
||||||
|
|
||||||
|
def item_title(self, item):
|
||||||
|
"""Generate item title: date + nick"""
|
||||||
|
# Format: "2025-11-15 - username"
|
||||||
|
date_str = item.created_at.strftime("%Y-%m-%d")
|
||||||
|
return f"{date_str} - {item.profile.nick}"
|
||||||
|
|
||||||
|
def item_description(self, item):
|
||||||
|
"""Return full post content as description, converted from Org to HTML"""
|
||||||
|
if not item.content:
|
||||||
|
return ""
|
||||||
|
|
||||||
|
# Convert Org mode content to HTML if org-python is available
|
||||||
|
if HAS_ORGPYTHON:
|
||||||
|
try:
|
||||||
|
# Posts in Org Social start at level 3 (***), but in RSS context
|
||||||
|
# they should start at level 1. Remove 2 asterisks from headings
|
||||||
|
# before conversion: *** -> *, **** -> **, etc.
|
||||||
|
content = re.sub(
|
||||||
|
r"^(\*{2,})",
|
||||||
|
lambda m: "*" * (len(m.group(1)) - 2),
|
||||||
|
item.content,
|
||||||
|
flags=re.MULTILINE,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Convert org-mode to HTML
|
||||||
|
# We disable toc as we're showing individual posts
|
||||||
|
# highlight=True enables syntax highlighting for code blocks
|
||||||
|
html_content = org_to_html(content, toc=False, highlight=True)
|
||||||
|
|
||||||
|
return html_content
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(
|
||||||
|
f"Failed to convert Org to HTML for post {item.post_id}: {e}"
|
||||||
|
)
|
||||||
|
# Fallback to plain text
|
||||||
|
return item.content
|
||||||
|
|
||||||
|
# If org-python is not available, return plain text
|
||||||
|
return item.content
|
||||||
|
|
||||||
|
def item_link(self, item):
|
||||||
|
"""Generate item link (post URL)"""
|
||||||
|
return f"{item.profile.feed}#{item.post_id}"
|
||||||
|
|
||||||
|
def item_author_name(self, item):
|
||||||
|
"""Return author name"""
|
||||||
|
return item.profile.nick
|
||||||
|
|
||||||
|
def item_author_email(self, item):
|
||||||
|
"""Return author email - using feed URL as identifier"""
|
||||||
|
# RSS 2.0 requires email format for author, but we don't have emails
|
||||||
|
# We'll return None and handle it differently
|
||||||
|
return None
|
||||||
|
|
||||||
|
def item_extra_kwargs(self, item):
|
||||||
|
"""Add custom author field"""
|
||||||
|
return {"author": item.profile.nick}
|
||||||
|
|
||||||
|
def item_pubdate(self, item):
|
||||||
|
"""Return publication date"""
|
||||||
|
return item.created_at
|
||||||
|
|
||||||
|
def item_categories(self, item):
|
||||||
|
"""Return post tags as categories"""
|
||||||
|
if item.tags:
|
||||||
|
return item.tags.split()
|
||||||
|
return []
|
||||||
|
|
||||||
|
def item_guid(self, item):
|
||||||
|
"""Return unique identifier for the item"""
|
||||||
|
return f"{item.profile.feed}#{item.post_id}"
|
||||||
|
|
||||||
|
def item_guid_is_permalink(self, item):
|
||||||
|
"""The GUID is a permalink"""
|
||||||
|
return True
|
||||||
1
app/sse_notifications/__init__.py
Normal file
1
app/sse_notifications/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
# SSE Notifications app
|
||||||
6
app/sse_notifications/apps.py
Normal file
6
app/sse_notifications/apps.py
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
from django.apps import AppConfig
|
||||||
|
|
||||||
|
|
||||||
|
class SseNotificationsConfig(AppConfig):
|
||||||
|
default_auto_field = "django.db.models.BigAutoField"
|
||||||
|
name = "app.sse_notifications"
|
||||||
359
app/sse_notifications/tests.py
Normal file
359
app/sse_notifications/tests.py
Normal file
|
|
@ -0,0 +1,359 @@
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import pytest
|
||||||
|
from unittest.mock import patch, AsyncMock, MagicMock
|
||||||
|
from django.test import TestCase, Client
|
||||||
|
from app.feeds.notification_publisher import publish_notification
|
||||||
|
|
||||||
|
|
||||||
|
def collect_stream(response):
|
||||||
|
"""Consume an async streaming response and return the decoded content."""
|
||||||
|
async def _collect():
|
||||||
|
chunks = []
|
||||||
|
async for chunk in response.streaming_content:
|
||||||
|
if isinstance(chunk, bytes):
|
||||||
|
chunks.append(chunk)
|
||||||
|
else:
|
||||||
|
chunks.append(chunk.encode("utf-8"))
|
||||||
|
return b"".join(chunks).decode("utf-8")
|
||||||
|
return asyncio.run(_collect())
|
||||||
|
|
||||||
|
|
||||||
|
def make_async_pubsub(messages=None, error=None):
|
||||||
|
"""Build an async pubsub mock that returns messages then stops."""
|
||||||
|
mock_pubsub = AsyncMock()
|
||||||
|
|
||||||
|
if error:
|
||||||
|
mock_pubsub.get_message.side_effect = error
|
||||||
|
elif messages is not None:
|
||||||
|
# Yield each message, then raise RedisError to break the loop
|
||||||
|
import redis.asyncio as aioredis
|
||||||
|
mock_pubsub.get_message.side_effect = messages + [aioredis.RedisError("done")]
|
||||||
|
else:
|
||||||
|
import redis.asyncio as aioredis
|
||||||
|
mock_pubsub.get_message.side_effect = aioredis.RedisError("done")
|
||||||
|
|
||||||
|
return mock_pubsub
|
||||||
|
|
||||||
|
|
||||||
|
class TestSSENotificationsEndpoint(TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
"""Set up test fixtures."""
|
||||||
|
self.client = Client()
|
||||||
|
self.feed_url = "https://example.com/social.org"
|
||||||
|
|
||||||
|
def _patch_redis(self, pubsub):
|
||||||
|
mock_redis = AsyncMock()
|
||||||
|
mock_redis.pubsub = MagicMock(return_value=pubsub) # pubsub() is a sync call
|
||||||
|
return patch("app.sse_notifications.views.aioredis.Redis", return_value=mock_redis)
|
||||||
|
|
||||||
|
def test_sse_endpoint_without_feed_returns_global_stream(self):
|
||||||
|
"""Test that SSE endpoint without feed parameter returns global stream."""
|
||||||
|
# Given: A Redis pubsub with no pending messages
|
||||||
|
pubsub = make_async_pubsub()
|
||||||
|
|
||||||
|
with self._patch_redis(pubsub):
|
||||||
|
# When: The SSE endpoint is requested without a feed parameter
|
||||||
|
response = self.client.get("/sse/notifications/")
|
||||||
|
|
||||||
|
# Then: An event stream response is returned
|
||||||
|
self.assertEqual(response.status_code, 200)
|
||||||
|
self.assertEqual(response["Content-Type"], "text/event-stream")
|
||||||
|
|
||||||
|
def test_sse_endpoint_accepts_valid_feed(self):
|
||||||
|
"""Test that SSE endpoint accepts valid feed parameter."""
|
||||||
|
# Given: A Redis pubsub with no pending messages
|
||||||
|
pubsub = make_async_pubsub()
|
||||||
|
|
||||||
|
with self._patch_redis(pubsub):
|
||||||
|
# When: The SSE endpoint is requested for a specific feed
|
||||||
|
response = self.client.get("/sse/notifications/", {"feed": self.feed_url})
|
||||||
|
|
||||||
|
# Then: An event stream response is returned with anti-buffering headers
|
||||||
|
self.assertEqual(response.status_code, 200)
|
||||||
|
self.assertEqual(response["Content-Type"], "text/event-stream")
|
||||||
|
self.assertEqual(response["Cache-Control"], "no-cache")
|
||||||
|
self.assertEqual(response["X-Accel-Buffering"], "no")
|
||||||
|
self.assertEqual(response["Access-Control-Allow-Origin"], "*")
|
||||||
|
|
||||||
|
def test_sse_sends_connection_event_with_feed(self):
|
||||||
|
"""Test that per-feed SSE sends initial connection event with feed field."""
|
||||||
|
# Given: A Redis pubsub with no pending messages
|
||||||
|
pubsub = make_async_pubsub()
|
||||||
|
|
||||||
|
with self._patch_redis(pubsub):
|
||||||
|
# When: The per-feed stream is requested and consumed
|
||||||
|
response = self.client.get("/sse/notifications/", {"feed": self.feed_url})
|
||||||
|
content = collect_stream(response)
|
||||||
|
|
||||||
|
# Then: The stream starts with a connection event naming the feed
|
||||||
|
self.assertIn("event: connected", content)
|
||||||
|
self.assertIn(f'"feed": "{self.feed_url}"', content)
|
||||||
|
self.assertIn('"status": "connected"', content)
|
||||||
|
|
||||||
|
def test_sse_sends_connection_event_global(self):
|
||||||
|
"""Test that global SSE sends initial connection event without feed field."""
|
||||||
|
# Given: A Redis pubsub with no pending messages
|
||||||
|
pubsub = make_async_pubsub()
|
||||||
|
|
||||||
|
with self._patch_redis(pubsub):
|
||||||
|
# When: The global stream is requested and consumed
|
||||||
|
response = self.client.get("/sse/notifications/")
|
||||||
|
content = collect_stream(response)
|
||||||
|
|
||||||
|
# Then: The stream starts with a connection event without a feed field
|
||||||
|
self.assertIn("event: connected", content)
|
||||||
|
self.assertIn('"status": "connected"', content)
|
||||||
|
self.assertNotIn('"feed":', content)
|
||||||
|
|
||||||
|
def test_sse_receives_notification_from_redis(self):
|
||||||
|
"""Test that per-feed SSE receives and forwards notifications from Redis."""
|
||||||
|
# Given: A Redis pubsub holding one mention notification
|
||||||
|
notification_data = {
|
||||||
|
"type": "mention",
|
||||||
|
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
|
||||||
|
}
|
||||||
|
pubsub = make_async_pubsub(
|
||||||
|
messages=[{"type": "message", "data": json.dumps(notification_data)}]
|
||||||
|
)
|
||||||
|
|
||||||
|
with self._patch_redis(pubsub):
|
||||||
|
# When: The per-feed stream is requested and consumed
|
||||||
|
response = self.client.get("/sse/notifications/", {"feed": self.feed_url})
|
||||||
|
content = collect_stream(response)
|
||||||
|
|
||||||
|
# Then: The notification is forwarded as an SSE event
|
||||||
|
self.assertIn("event: notification", content)
|
||||||
|
self.assertIn('"type": "mention"', content)
|
||||||
|
self.assertIn("https://alice.com/social.org#2024-01-01T10:00:00+0000", content)
|
||||||
|
|
||||||
|
def test_global_sse_adds_target_feed_to_notifications(self):
|
||||||
|
"""Test that global SSE adds target_feed field extracted from channel name."""
|
||||||
|
# Given: A Redis pubsub holding one pattern message for a feed channel
|
||||||
|
notification_data = {
|
||||||
|
"type": "mention",
|
||||||
|
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
|
||||||
|
}
|
||||||
|
target_feed = "https://example.com/social.org"
|
||||||
|
pubsub = make_async_pubsub(
|
||||||
|
messages=[
|
||||||
|
{
|
||||||
|
"type": "pmessage",
|
||||||
|
"pattern": "notifications:*",
|
||||||
|
"channel": f"notifications:{target_feed}",
|
||||||
|
"data": json.dumps(notification_data),
|
||||||
|
}
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
with self._patch_redis(pubsub):
|
||||||
|
# When: The global stream is requested and consumed
|
||||||
|
response = self.client.get("/sse/notifications/")
|
||||||
|
content = collect_stream(response)
|
||||||
|
|
||||||
|
# Then: The forwarded event includes the target_feed from the channel
|
||||||
|
self.assertIn("event: notification", content)
|
||||||
|
self.assertIn(f'"target_feed": "{target_feed}"', content)
|
||||||
|
self.assertIn('"type": "mention"', content)
|
||||||
|
|
||||||
|
def test_global_sse_uses_psubscribe(self):
|
||||||
|
"""Test that global SSE subscribes with psubscribe to all notification channels."""
|
||||||
|
# Given: A Redis pubsub with no pending messages
|
||||||
|
pubsub = make_async_pubsub()
|
||||||
|
|
||||||
|
with self._patch_redis(pubsub):
|
||||||
|
# When: The global stream is requested and consumed
|
||||||
|
response = self.client.get("/sse/notifications/")
|
||||||
|
collect_stream(response)
|
||||||
|
|
||||||
|
# Then: The view subscribed by pattern to every notification channel
|
||||||
|
pubsub.psubscribe.assert_awaited_once_with("notifications:*")
|
||||||
|
|
||||||
|
def test_per_feed_sse_uses_subscribe(self):
|
||||||
|
"""Test that per-feed SSE subscribes with subscribe to the specific channel."""
|
||||||
|
# Given: A Redis pubsub with no pending messages
|
||||||
|
pubsub = make_async_pubsub()
|
||||||
|
|
||||||
|
with self._patch_redis(pubsub):
|
||||||
|
# When: The per-feed stream is requested and consumed
|
||||||
|
response = self.client.get("/sse/notifications/", {"feed": self.feed_url})
|
||||||
|
collect_stream(response)
|
||||||
|
|
||||||
|
# Then: The view subscribed only to that feed's channel
|
||||||
|
pubsub.subscribe.assert_awaited_once_with(f"notifications:{self.feed_url}")
|
||||||
|
|
||||||
|
|
||||||
|
class TestNotificationPublisher(TestCase):
|
||||||
|
"""Test the notification publisher module"""
|
||||||
|
|
||||||
|
@patch("app.feeds.notification_publisher.redis.Redis")
|
||||||
|
def test_publish_mention_notification(self, mock_redis):
|
||||||
|
"""Test publishing a mention notification"""
|
||||||
|
# Given: A working Redis connection and a mention to notify
|
||||||
|
mock_redis_instance = MagicMock()
|
||||||
|
mock_redis.return_value = mock_redis_instance
|
||||||
|
|
||||||
|
target_feed = "https://bob.com/social.org"
|
||||||
|
post_url = "https://alice.com/social.org#2024-01-01T10:00:00+0000"
|
||||||
|
|
||||||
|
# When: The mention notification is published
|
||||||
|
result = publish_notification(
|
||||||
|
target_feed_url=target_feed, notification_type="mention", post_url=post_url
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: One message is published on the target feed's channel
|
||||||
|
self.assertTrue(result)
|
||||||
|
mock_redis_instance.publish.assert_called_once()
|
||||||
|
|
||||||
|
call_args = mock_redis_instance.publish.call_args
|
||||||
|
channel, data = call_args[0]
|
||||||
|
self.assertEqual(channel, f"notifications:{target_feed}")
|
||||||
|
notification = json.loads(data)
|
||||||
|
self.assertEqual(notification["type"], "mention")
|
||||||
|
self.assertEqual(notification["post"], post_url)
|
||||||
|
|
||||||
|
@patch("app.feeds.notification_publisher.redis.Redis")
|
||||||
|
def test_publish_reaction_notification_with_emoji(self, mock_redis):
|
||||||
|
"""Test publishing a reaction notification with emoji"""
|
||||||
|
# Given: A working Redis connection and a reaction to notify
|
||||||
|
mock_redis_instance = MagicMock()
|
||||||
|
mock_redis.return_value = mock_redis_instance
|
||||||
|
|
||||||
|
# When: The reaction notification is published
|
||||||
|
result = publish_notification(
|
||||||
|
target_feed_url="https://bob.com/social.org",
|
||||||
|
notification_type="reaction",
|
||||||
|
post_url="https://alice.com/social.org#2024-01-01T10:00:00+0000",
|
||||||
|
emoji="❤",
|
||||||
|
parent="https://bob.com/social.org#2024-01-01T09:00:00+0000",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: The published message carries the reaction type and emoji
|
||||||
|
self.assertTrue(result)
|
||||||
|
call_args = mock_redis_instance.publish.call_args
|
||||||
|
notification = json.loads(call_args[0][1])
|
||||||
|
self.assertEqual(notification["type"], "reaction")
|
||||||
|
self.assertEqual(notification["emoji"], "❤")
|
||||||
|
|
||||||
|
@patch("app.feeds.notification_publisher.redis.Redis")
|
||||||
|
def test_publish_reply_notification(self, mock_redis):
|
||||||
|
"""Test publishing a reply notification"""
|
||||||
|
# Given: A working Redis connection and a reply to notify
|
||||||
|
mock_redis_instance = MagicMock()
|
||||||
|
mock_redis.return_value = mock_redis_instance
|
||||||
|
|
||||||
|
# When: The reply notification is published
|
||||||
|
result = publish_notification(
|
||||||
|
target_feed_url="https://bob.com/social.org",
|
||||||
|
notification_type="reply",
|
||||||
|
post_url="https://alice.com/social.org#2024-01-01T10:00:00+0000",
|
||||||
|
parent="https://bob.com/social.org#2024-01-01T09:00:00+0000",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: The published message carries the reply type
|
||||||
|
self.assertTrue(result)
|
||||||
|
notification = json.loads(mock_redis_instance.publish.call_args[0][1])
|
||||||
|
self.assertEqual(notification["type"], "reply")
|
||||||
|
|
||||||
|
@patch("app.feeds.notification_publisher.redis.Redis")
|
||||||
|
def test_publish_boost_notification(self, mock_redis):
|
||||||
|
"""Test publishing a boost notification"""
|
||||||
|
# Given: A working Redis connection and a boost to notify
|
||||||
|
mock_redis_instance = MagicMock()
|
||||||
|
mock_redis.return_value = mock_redis_instance
|
||||||
|
|
||||||
|
# When: The boost notification is published
|
||||||
|
result = publish_notification(
|
||||||
|
target_feed_url="https://bob.com/social.org",
|
||||||
|
notification_type="boost",
|
||||||
|
post_url="https://alice.com/social.org#2024-01-01T10:00:00+0000",
|
||||||
|
boosted="https://bob.com/social.org#2024-01-01T09:00:00+0000",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: The published message carries the boost type
|
||||||
|
self.assertTrue(result)
|
||||||
|
notification = json.loads(mock_redis_instance.publish.call_args[0][1])
|
||||||
|
self.assertEqual(notification["type"], "boost")
|
||||||
|
|
||||||
|
@patch("app.feeds.notification_publisher.redis.Redis")
|
||||||
|
def test_publish_notification_handles_redis_error(self, mock_redis):
|
||||||
|
"""Test that publish_notification handles Redis errors gracefully"""
|
||||||
|
# Given: A Redis connection that fails
|
||||||
|
mock_redis.side_effect = Exception("Redis connection failed")
|
||||||
|
|
||||||
|
# When: A notification is published
|
||||||
|
result = publish_notification(
|
||||||
|
target_feed_url="https://bob.com/social.org",
|
||||||
|
notification_type="mention",
|
||||||
|
post_url="https://alice.com/social.org#2024-01-01T10:00:00+0000",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: The failure is reported without raising
|
||||||
|
self.assertFalse(result)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.django_db
|
||||||
|
class TestSSENotificationStructure:
|
||||||
|
"""Test that SSE notifications match the expected JSON structure"""
|
||||||
|
|
||||||
|
def test_mention_notification_structure(self):
|
||||||
|
# Given: A mention notification payload
|
||||||
|
notification = {
|
||||||
|
"type": "mention",
|
||||||
|
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Then: It carries the mention type and a post URL with fragment
|
||||||
|
assert "type" in notification
|
||||||
|
assert notification["type"] == "mention"
|
||||||
|
assert "#" in notification["post"]
|
||||||
|
|
||||||
|
def test_reaction_notification_structure(self):
|
||||||
|
# Given: A reaction notification payload
|
||||||
|
notification = {
|
||||||
|
"type": "reaction",
|
||||||
|
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
|
||||||
|
"emoji": "❤",
|
||||||
|
"parent": "https://bob.com/social.org#2024-01-01T09:00:00+0000",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Then: It carries the reaction type, emoji and parent
|
||||||
|
assert notification["type"] == "reaction"
|
||||||
|
assert "emoji" in notification
|
||||||
|
assert "parent" in notification
|
||||||
|
|
||||||
|
def test_reply_notification_structure(self):
|
||||||
|
# Given: A reply notification payload
|
||||||
|
notification = {
|
||||||
|
"type": "reply",
|
||||||
|
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
|
||||||
|
"parent": "https://bob.com/social.org#2024-01-01T09:00:00+0000",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Then: It carries the reply type and parent
|
||||||
|
assert notification["type"] == "reply"
|
||||||
|
assert "parent" in notification
|
||||||
|
|
||||||
|
def test_boost_notification_structure(self):
|
||||||
|
# Given: A boost notification payload
|
||||||
|
notification = {
|
||||||
|
"type": "boost",
|
||||||
|
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
|
||||||
|
"boosted": "https://bob.com/social.org#2024-01-01T09:00:00+0000",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Then: It carries the boost type and the boosted post
|
||||||
|
assert notification["type"] == "boost"
|
||||||
|
assert "boosted" in notification
|
||||||
|
|
||||||
|
def test_global_notification_includes_target_feed(self):
|
||||||
|
# Given: A global stream notification payload
|
||||||
|
notification = {
|
||||||
|
"target_feed": "https://example.com/social.org",
|
||||||
|
"type": "mention",
|
||||||
|
"post": "https://alice.com/social.org#2024-01-01T10:00:00+0000",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Then: It includes the target feed as an absolute URL
|
||||||
|
assert "target_feed" in notification
|
||||||
|
assert notification["target_feed"].startswith("http")
|
||||||
6
app/sse_notifications/urls.py
Normal file
6
app/sse_notifications/urls.py
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
from django.urls import path
|
||||||
|
from .views import SSENotificationsView
|
||||||
|
|
||||||
|
urlpatterns = [
|
||||||
|
path("notifications/", SSENotificationsView.as_view(), name="sse-notifications"),
|
||||||
|
]
|
||||||
144
app/sse_notifications/views.py
Normal file
144
app/sse_notifications/views.py
Normal file
|
|
@ -0,0 +1,144 @@
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
import logging
|
||||||
|
from django.http import StreamingHttpResponse
|
||||||
|
from django.views import View
|
||||||
|
from django.conf import settings
|
||||||
|
|
||||||
|
import redis.asyncio as aioredis
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
HEARTBEAT_INTERVAL = 30
|
||||||
|
|
||||||
|
|
||||||
|
def _sse_response(stream):
|
||||||
|
response = StreamingHttpResponse(stream, content_type="text/event-stream")
|
||||||
|
response["Cache-Control"] = "no-cache"
|
||||||
|
response["X-Accel-Buffering"] = "no"
|
||||||
|
response["Access-Control-Allow-Origin"] = "*"
|
||||||
|
response["Access-Control-Allow-Methods"] = "GET, OPTIONS"
|
||||||
|
response["Access-Control-Allow-Headers"] = "Content-Type"
|
||||||
|
return response
|
||||||
|
|
||||||
|
|
||||||
|
def _get_redis():
|
||||||
|
host = settings.HUEY["connection"]["host"]
|
||||||
|
port = settings.HUEY["connection"]["port"]
|
||||||
|
db = settings.HUEY["connection"]["db"]
|
||||||
|
return aioredis.Redis(host=host, port=port, db=db, decode_responses=True)
|
||||||
|
|
||||||
|
|
||||||
|
class SSENotificationsView(View):
|
||||||
|
"""
|
||||||
|
Server-Sent Events endpoint for real-time notifications.
|
||||||
|
|
||||||
|
With ?feed=: streams notifications for a single feed.
|
||||||
|
Without ?feed=: streams all notifications from all feeds, adding target_feed to each event.
|
||||||
|
"""
|
||||||
|
|
||||||
|
async def get(self, request):
|
||||||
|
feed_url = request.GET.get("feed", "").strip()
|
||||||
|
stream = self._feed_stream(feed_url) if feed_url else self._global_stream()
|
||||||
|
return _sse_response(stream)
|
||||||
|
|
||||||
|
async def _feed_stream(self, feed_url):
|
||||||
|
r = _get_redis()
|
||||||
|
pubsub = r.pubsub()
|
||||||
|
logger.info(f"SSE per-feed connection: {feed_url}")
|
||||||
|
try:
|
||||||
|
await pubsub.subscribe(f"notifications:{feed_url}")
|
||||||
|
|
||||||
|
yield "event: connected\n"
|
||||||
|
yield f"data: {json.dumps({'feed': feed_url, 'status': 'connected'})}\n\n"
|
||||||
|
|
||||||
|
async for chunk in self._message_loop(pubsub):
|
||||||
|
yield chunk
|
||||||
|
|
||||||
|
except aioredis.RedisError as e:
|
||||||
|
logger.error(f"Redis error for {feed_url}: {e}")
|
||||||
|
yield "event: error\n"
|
||||||
|
yield f"data: {json.dumps({'error': 'Redis connection failed'})}\n\n"
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Unexpected error in SSE stream for {feed_url}: {e}")
|
||||||
|
yield "event: error\n"
|
||||||
|
yield f"data: {json.dumps({'error': 'Internal server error'})}\n\n"
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
await pubsub.aclose()
|
||||||
|
await r.aclose()
|
||||||
|
logger.info(f"SSE connection closed for feed: {feed_url}")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
async def _global_stream(self):
|
||||||
|
r = _get_redis()
|
||||||
|
pubsub = r.pubsub()
|
||||||
|
logger.info("SSE global connection established")
|
||||||
|
try:
|
||||||
|
await pubsub.psubscribe("notifications:*")
|
||||||
|
|
||||||
|
yield "event: connected\n"
|
||||||
|
yield f"data: {json.dumps({'status': 'connected'})}\n\n"
|
||||||
|
|
||||||
|
async for chunk in self._message_loop(pubsub, global_mode=True):
|
||||||
|
yield chunk
|
||||||
|
|
||||||
|
except aioredis.RedisError as e:
|
||||||
|
logger.error(f"Redis error in global SSE stream: {e}")
|
||||||
|
yield "event: error\n"
|
||||||
|
yield f"data: {json.dumps({'error': 'Redis connection failed'})}\n\n"
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Unexpected error in global SSE stream: {e}")
|
||||||
|
yield "event: error\n"
|
||||||
|
yield f"data: {json.dumps({'error': 'Internal server error'})}\n\n"
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
await pubsub.aclose()
|
||||||
|
await r.aclose()
|
||||||
|
logger.info("Global SSE connection closed")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
async def _message_loop(self, pubsub, global_mode=False):
|
||||||
|
last_heartbeat = time.time()
|
||||||
|
|
||||||
|
while True:
|
||||||
|
current_time = time.time()
|
||||||
|
if current_time - last_heartbeat >= HEARTBEAT_INTERVAL:
|
||||||
|
yield "event: heartbeat\n"
|
||||||
|
yield f"data: {json.dumps({'status': 'alive', 'timestamp': int(current_time)})}\n\n"
|
||||||
|
last_heartbeat = current_time
|
||||||
|
|
||||||
|
message = await pubsub.get_message(
|
||||||
|
ignore_subscribe_messages=True, timeout=1.0
|
||||||
|
)
|
||||||
|
|
||||||
|
if message is None:
|
||||||
|
await asyncio.sleep(0)
|
||||||
|
continue
|
||||||
|
|
||||||
|
msg_type = message["type"]
|
||||||
|
is_data = (msg_type == "message") or (
|
||||||
|
global_mode and msg_type == "pmessage"
|
||||||
|
)
|
||||||
|
if not is_data:
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
notification_data = json.loads(message["data"])
|
||||||
|
|
||||||
|
if global_mode:
|
||||||
|
channel = message["channel"]
|
||||||
|
notification_data["target_feed"] = channel.removeprefix(
|
||||||
|
"notifications:"
|
||||||
|
)
|
||||||
|
|
||||||
|
yield "event: notification\n"
|
||||||
|
yield f"data: {json.dumps(notification_data)}\n\n"
|
||||||
|
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
logger.error(f"Failed to decode notification message: {e}")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error processing notification: {e}")
|
||||||
0
app/stats/__init__.py
Normal file
0
app/stats/__init__.py
Normal file
6
app/stats/apps.py
Normal file
6
app/stats/apps.py
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
from django.apps import AppConfig
|
||||||
|
|
||||||
|
|
||||||
|
class StatsConfig(AppConfig):
|
||||||
|
default_auto_field = "django.db.models.BigAutoField"
|
||||||
|
name = "app.stats"
|
||||||
295
app/stats/tests.py
Normal file
295
app/stats/tests.py
Normal file
|
|
@ -0,0 +1,295 @@
|
||||||
|
from datetime import datetime, timezone as dt_timezone
|
||||||
|
|
||||||
|
from django.core.cache import cache
|
||||||
|
from django.test import TestCase, override_settings
|
||||||
|
from rest_framework import status
|
||||||
|
from rest_framework.test import APIClient
|
||||||
|
|
||||||
|
from app.feeds.models import Feed, Follow, Post, Profile
|
||||||
|
|
||||||
|
|
||||||
|
class StatsViewTest(TestCase):
|
||||||
|
"""Test cases for the StatsView API using Given/When/Then structure."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.client = APIClient()
|
||||||
|
self.stats_url = "/stats/"
|
||||||
|
cache.clear()
|
||||||
|
|
||||||
|
self.profile1 = Profile.objects.create(
|
||||||
|
feed="https://alice.org/social.org",
|
||||||
|
title="Alice",
|
||||||
|
nick="alice",
|
||||||
|
)
|
||||||
|
self.profile2 = Profile.objects.create(
|
||||||
|
feed="https://bob.org/social.org",
|
||||||
|
title="Bob",
|
||||||
|
nick="bob",
|
||||||
|
)
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
cache.clear()
|
||||||
|
|
||||||
|
def _create_post(self, profile, post_id, **kwargs):
|
||||||
|
"""Create a post whose created_at matches its RFC 3339 post_id"""
|
||||||
|
created_at = datetime.fromisoformat(post_id).astimezone(dt_timezone.utc)
|
||||||
|
return Post.objects.create(
|
||||||
|
profile=profile,
|
||||||
|
post_id=post_id,
|
||||||
|
created_at=created_at,
|
||||||
|
**kwargs,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_empty_database_returns_empty_stats(self):
|
||||||
|
"""Test GET /stats/ with no posts returns empty years and zeroed globals."""
|
||||||
|
# Given: No posts, feeds or follows exist (only two profiles)
|
||||||
|
|
||||||
|
# When: We request the stats
|
||||||
|
response = self.client.get(self.stats_url)
|
||||||
|
|
||||||
|
# Then: The response is successful with empty monthly data
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
self.assertEqual(response.data["type"], "Success")
|
||||||
|
self.assertEqual(response.data["errors"], [])
|
||||||
|
self.assertEqual(response.data["data"]["years"], {})
|
||||||
|
self.assertEqual(
|
||||||
|
response.data["data"]["global"],
|
||||||
|
{
|
||||||
|
"registered_feeds": 0,
|
||||||
|
"total_accounts": 2,
|
||||||
|
"total_posts": 0,
|
||||||
|
"total_follows": 0,
|
||||||
|
"active_groups": 0,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
self.assertIn("generated_at", response.data["meta"])
|
||||||
|
self.assertEqual(
|
||||||
|
response.data["_links"]["self"],
|
||||||
|
{"href": "/stats/", "method": "GET"},
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_post_types_are_disjoint_and_sum_to_total(self):
|
||||||
|
"""Test each post type is counted once and the sum matches total_posts."""
|
||||||
|
# Given: One post of each type in the same month
|
||||||
|
self._create_post(self.profile1, "2025-03-01T10:00:00+00:00", content="Plain")
|
||||||
|
self._create_post(
|
||||||
|
self.profile1,
|
||||||
|
"2025-03-02T10:00:00+00:00",
|
||||||
|
content="A reply with text",
|
||||||
|
reply_to="https://bob.org/social.org#2025-03-01T09:00:00+00:00",
|
||||||
|
)
|
||||||
|
self._create_post(
|
||||||
|
self.profile1,
|
||||||
|
"2025-03-03T10:00:00+00:00",
|
||||||
|
content="",
|
||||||
|
reply_to="https://bob.org/social.org#2025-03-01T09:00:00+00:00",
|
||||||
|
mood="😄",
|
||||||
|
)
|
||||||
|
self._create_post(
|
||||||
|
self.profile1,
|
||||||
|
"2025-03-04T10:00:00+00:00",
|
||||||
|
content="",
|
||||||
|
include="https://bob.org/social.org#2025-03-01T09:00:00+00:00",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: We request the stats
|
||||||
|
response = self.client.get(self.stats_url)
|
||||||
|
|
||||||
|
# Then: Each type is counted once and the four counters sum to total
|
||||||
|
month = response.data["data"]["years"]["2025"]["03"]
|
||||||
|
self.assertEqual(month["total_posts"], 4)
|
||||||
|
self.assertEqual(month["posts"], 1)
|
||||||
|
self.assertEqual(month["replies"], 1)
|
||||||
|
self.assertEqual(month["reactions"], 1)
|
||||||
|
self.assertEqual(month["boosts"], 1)
|
||||||
|
self.assertEqual(
|
||||||
|
month["posts"] + month["replies"] + month["reactions"] + month["boosts"],
|
||||||
|
month["total_posts"],
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_reaction_requires_mood_and_empty_content(self):
|
||||||
|
"""Test replies with mood and text count as replies, not reactions."""
|
||||||
|
# Given: A reply with mood but real content, and a reaction with
|
||||||
|
# whitespace-only content
|
||||||
|
self._create_post(
|
||||||
|
self.profile1,
|
||||||
|
"2025-05-01T10:00:00+00:00",
|
||||||
|
content="Great post!",
|
||||||
|
reply_to="https://bob.org/social.org#2025-04-30T09:00:00+00:00",
|
||||||
|
mood="🔥",
|
||||||
|
)
|
||||||
|
self._create_post(
|
||||||
|
self.profile2,
|
||||||
|
"2025-05-02T10:00:00+00:00",
|
||||||
|
content=" \n ",
|
||||||
|
reply_to="https://alice.org/social.org#2025-04-30T09:00:00+00:00",
|
||||||
|
mood="👍",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: We request the stats
|
||||||
|
response = self.client.get(self.stats_url)
|
||||||
|
|
||||||
|
# Then: Only the empty-content reply counts as a reaction
|
||||||
|
month = response.data["data"]["years"]["2025"]["05"]
|
||||||
|
self.assertEqual(month["replies"], 1)
|
||||||
|
self.assertEqual(month["reactions"], 1)
|
||||||
|
|
||||||
|
def test_group_messages_and_polls_are_transversal(self):
|
||||||
|
"""Test group messages and polls also count in their base type."""
|
||||||
|
# Given: A poll and a reply inside a group
|
||||||
|
self._create_post(
|
||||||
|
self.profile1,
|
||||||
|
"2025-06-01T10:00:00+00:00",
|
||||||
|
content="Cat or dog?",
|
||||||
|
poll_end=datetime(2025, 6, 10, tzinfo=dt_timezone.utc),
|
||||||
|
)
|
||||||
|
self._create_post(
|
||||||
|
self.profile2,
|
||||||
|
"2025-06-02T10:00:00+00:00",
|
||||||
|
content="I agree",
|
||||||
|
reply_to="https://alice.org/social.org#2025-06-01T09:00:00+00:00",
|
||||||
|
group="emacs",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: We request the stats
|
||||||
|
response = self.client.get(self.stats_url)
|
||||||
|
|
||||||
|
# Then: The poll counts as post and the group reply counts as reply
|
||||||
|
month = response.data["data"]["years"]["2025"]["06"]
|
||||||
|
self.assertEqual(month["total_posts"], 2)
|
||||||
|
self.assertEqual(month["posts"], 1)
|
||||||
|
self.assertEqual(month["replies"], 1)
|
||||||
|
self.assertEqual(month["polls"], 1)
|
||||||
|
self.assertEqual(month["group_messages"], 1)
|
||||||
|
|
||||||
|
def test_active_accounts_counts_distinct_profiles(self):
|
||||||
|
"""Test active_accounts counts each profile once per month."""
|
||||||
|
# Given: Two posts from the same profile and one from another
|
||||||
|
self._create_post(self.profile1, "2025-07-01T10:00:00+00:00", content="One")
|
||||||
|
self._create_post(self.profile1, "2025-07-02T10:00:00+00:00", content="Two")
|
||||||
|
self._create_post(self.profile2, "2025-07-03T10:00:00+00:00", content="Three")
|
||||||
|
|
||||||
|
# When: We request the stats
|
||||||
|
response = self.client.get(self.stats_url)
|
||||||
|
|
||||||
|
# Then: Only two distinct accounts are active
|
||||||
|
month = response.data["data"]["years"]["2025"]["07"]
|
||||||
|
self.assertEqual(month["active_accounts"], 2)
|
||||||
|
self.assertEqual(month["total_posts"], 3)
|
||||||
|
|
||||||
|
def test_posts_are_grouped_by_year_and_month(self):
|
||||||
|
"""Test posts land in the year/month of their ID with padded keys."""
|
||||||
|
# Given: Posts in different months and years
|
||||||
|
self._create_post(self.profile1, "2024-12-15T10:00:00+00:00", content="Dec")
|
||||||
|
self._create_post(self.profile1, "2025-01-15T10:00:00+00:00", content="Jan")
|
||||||
|
self._create_post(self.profile2, "2025-01-20T10:00:00+00:00", content="Jan 2")
|
||||||
|
|
||||||
|
# When: We request the stats
|
||||||
|
response = self.client.get(self.stats_url)
|
||||||
|
|
||||||
|
# Then: Months are grouped under their year with zero-padded keys
|
||||||
|
years = response.data["data"]["years"]
|
||||||
|
self.assertEqual(sorted(years.keys()), ["2024", "2025"])
|
||||||
|
self.assertEqual(years["2024"]["12"]["total_posts"], 1)
|
||||||
|
self.assertEqual(years["2025"]["01"]["total_posts"], 2)
|
||||||
|
|
||||||
|
def test_global_counters(self):
|
||||||
|
"""Test global counters reflect feeds, accounts, follows and groups."""
|
||||||
|
# Given: Feeds, follows and posts in two different groups
|
||||||
|
Feed.objects.create(url="https://alice.org/social.org")
|
||||||
|
Feed.objects.create(url="https://bob.org/social.org")
|
||||||
|
Feed.objects.create(url="https://carol.org/social.org")
|
||||||
|
Follow.objects.create(follower=self.profile1, followed=self.profile2)
|
||||||
|
self._create_post(
|
||||||
|
self.profile1, "2025-08-01T10:00:00+00:00", content="Hi", group="emacs"
|
||||||
|
)
|
||||||
|
self._create_post(
|
||||||
|
self.profile1, "2025-08-02T10:00:00+00:00", content="Hi", group="emacs"
|
||||||
|
)
|
||||||
|
self._create_post(
|
||||||
|
self.profile2, "2025-08-03T10:00:00+00:00", content="Hi", group="org-mode"
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: We request the stats
|
||||||
|
response = self.client.get(self.stats_url)
|
||||||
|
|
||||||
|
# Then: Global counters match the created records
|
||||||
|
self.assertEqual(
|
||||||
|
response.data["data"]["global"],
|
||||||
|
{
|
||||||
|
"registered_feeds": 3,
|
||||||
|
"total_accounts": 2,
|
||||||
|
"total_posts": 3,
|
||||||
|
"total_follows": 1,
|
||||||
|
"active_groups": 2,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
@override_settings(
|
||||||
|
CACHES={
|
||||||
|
"default": {
|
||||||
|
"BACKEND": "django.core.cache.backends.locmem.LocMemCache",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
def test_response_is_cached(self):
|
||||||
|
"""Test the response is cached until the cache is cleared."""
|
||||||
|
# Given: A first request that populates the cache (DEBUG uses
|
||||||
|
# DummyCache, so a real in-memory backend is forced here)
|
||||||
|
self._create_post(self.profile1, "2025-09-01T10:00:00+00:00", content="One")
|
||||||
|
first = self.client.get(self.stats_url)
|
||||||
|
self.assertEqual(first.data["data"]["global"]["total_posts"], 1)
|
||||||
|
|
||||||
|
# When: A new post arrives without clearing the cache
|
||||||
|
self._create_post(self.profile1, "2025-09-02T10:00:00+00:00", content="Two")
|
||||||
|
cached = self.client.get(self.stats_url)
|
||||||
|
|
||||||
|
# Then: The cached response is returned until the cache is cleared
|
||||||
|
self.assertEqual(cached.data["data"]["global"]["total_posts"], 1)
|
||||||
|
cache.clear()
|
||||||
|
fresh = self.client.get(self.stats_url)
|
||||||
|
self.assertEqual(fresh.data["data"]["global"]["total_posts"], 2)
|
||||||
|
|
||||||
|
def test_bridge_feeds_do_not_count_in_stats(self):
|
||||||
|
"""Test bridge profiles, posts, feeds and follows are excluded."""
|
||||||
|
# Given: A real feed with posts and a bridge with feed, posts and follow
|
||||||
|
Feed.objects.create(url="https://alice.org/social.org")
|
||||||
|
self._create_post(self.profile1, "2025-08-01T10:00:00+00:00", content="Hi")
|
||||||
|
|
||||||
|
bridge_profile = Profile.objects.create(
|
||||||
|
feed=(
|
||||||
|
"https://relay.org-social.org/bridge/rss/"
|
||||||
|
"?url=https%3A%2F%2Frss.arxiv.org%2Frss%2Fquant-ph"
|
||||||
|
),
|
||||||
|
title="quant-ph updates on arXiv.org",
|
||||||
|
nick="quant-ph_updates_on_arXiv_org",
|
||||||
|
)
|
||||||
|
Feed.objects.create(url=bridge_profile.feed)
|
||||||
|
Follow.objects.create(follower=self.profile1, followed=bridge_profile)
|
||||||
|
for second in range(3):
|
||||||
|
self._create_post(
|
||||||
|
bridge_profile,
|
||||||
|
f"2025-08-02T04:00:0{second}+00:00",
|
||||||
|
content="Bridged paper",
|
||||||
|
group="physics",
|
||||||
|
)
|
||||||
|
|
||||||
|
# When: We request the stats
|
||||||
|
response = self.client.get(self.stats_url)
|
||||||
|
|
||||||
|
# Then: Global counters only reflect the real feed and its post
|
||||||
|
self.assertEqual(
|
||||||
|
response.data["data"]["global"],
|
||||||
|
{
|
||||||
|
"registered_feeds": 1,
|
||||||
|
"total_accounts": 2,
|
||||||
|
"total_posts": 1,
|
||||||
|
"total_follows": 0,
|
||||||
|
"active_groups": 0,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Then: The monthly breakdown ignores the bridged posts
|
||||||
|
month = response.data["data"]["years"]["2025"]["08"]
|
||||||
|
self.assertEqual(month["total_posts"], 1)
|
||||||
|
self.assertEqual(month["active_accounts"], 1)
|
||||||
8
app/stats/urls.py
Normal file
8
app/stats/urls.py
Normal file
|
|
@ -0,0 +1,8 @@
|
||||||
|
from django.urls import path
|
||||||
|
from .views import StatsView
|
||||||
|
|
||||||
|
app_name = "stats"
|
||||||
|
|
||||||
|
urlpatterns = [
|
||||||
|
path("", StatsView.as_view(), name="stats"),
|
||||||
|
]
|
||||||
117
app/stats/views.py
Normal file
117
app/stats/views.py
Normal file
|
|
@ -0,0 +1,117 @@
|
||||||
|
from rest_framework.views import APIView
|
||||||
|
from rest_framework.response import Response
|
||||||
|
from rest_framework import status
|
||||||
|
from django.core.cache import cache
|
||||||
|
from django.db.models import Count, Q
|
||||||
|
from django.db.models.functions import TruncMonth
|
||||||
|
from django.utils import timezone
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from app.bridge.models import bridge_urls_q
|
||||||
|
from app.feeds.models import Feed, Follow, Post, Profile
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class StatsView(APIView):
|
||||||
|
"""Aggregated statistics grouped by year and month, plus global counters"""
|
||||||
|
|
||||||
|
def get(self, request):
|
||||||
|
cache_key = "stats"
|
||||||
|
cached_response = cache.get(cache_key)
|
||||||
|
|
||||||
|
if cached_response is not None:
|
||||||
|
return Response(cached_response, status=status.HTTP_200_OK)
|
||||||
|
|
||||||
|
response_data = {
|
||||||
|
"type": "Success",
|
||||||
|
"errors": [],
|
||||||
|
"data": {
|
||||||
|
"years": self._get_monthly_stats(),
|
||||||
|
"global": self._get_global_stats(),
|
||||||
|
},
|
||||||
|
"meta": {
|
||||||
|
"generated_at": timezone.now().isoformat(),
|
||||||
|
},
|
||||||
|
"_links": {
|
||||||
|
"self": {"href": "/stats/", "method": "GET"},
|
||||||
|
"feeds": {"href": "/feeds/", "method": "GET"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
# Cache permanently (will be cleared by scan_feeds task)
|
||||||
|
cache.set(cache_key, response_data, None)
|
||||||
|
|
||||||
|
return Response(response_data, status=status.HTTP_200_OK)
|
||||||
|
|
||||||
|
def _get_monthly_stats(self):
|
||||||
|
"""Aggregate post counters grouped by year and month.
|
||||||
|
|
||||||
|
A post belongs to the month of its ID (RFC 3339 timestamp), which
|
||||||
|
is stored parsed in created_at. The four type counters are disjoint
|
||||||
|
(posts + replies + reactions + boosts = total_posts), while
|
||||||
|
group_messages and polls are transversal.
|
||||||
|
"""
|
||||||
|
is_reply = ~Q(reply_to="")
|
||||||
|
# Same criterion as the replies endpoint: a reaction is a reply with
|
||||||
|
# a mood and empty or whitespace-only content
|
||||||
|
is_reaction = is_reply & ~Q(mood="") & Q(content__regex=r"^\s*$")
|
||||||
|
|
||||||
|
# Bridged posts belong to external authors, not to real accounts
|
||||||
|
monthly = (
|
||||||
|
Post.objects.exclude(bridge_urls_q("profile__feed"))
|
||||||
|
.annotate(month=TruncMonth("created_at"))
|
||||||
|
.values("month")
|
||||||
|
.annotate(
|
||||||
|
active_accounts=Count("profile", distinct=True),
|
||||||
|
total_posts=Count("id"),
|
||||||
|
posts=Count("id", filter=Q(reply_to="") & Q(include="")),
|
||||||
|
replies=Count("id", filter=is_reply & ~is_reaction),
|
||||||
|
boosts=Count("id", filter=Q(reply_to="") & ~Q(include="")),
|
||||||
|
reactions=Count("id", filter=is_reaction),
|
||||||
|
group_messages=Count("id", filter=~Q(group="")),
|
||||||
|
polls=Count("id", filter=Q(poll_end__isnull=False)),
|
||||||
|
)
|
||||||
|
.order_by("month")
|
||||||
|
)
|
||||||
|
|
||||||
|
years = {}
|
||||||
|
for row in monthly:
|
||||||
|
month = row["month"]
|
||||||
|
if month is None:
|
||||||
|
continue
|
||||||
|
|
||||||
|
year_key = f"{month.year:04d}"
|
||||||
|
month_key = f"{month.month:02d}"
|
||||||
|
years.setdefault(year_key, {})[month_key] = {
|
||||||
|
"active_accounts": row["active_accounts"],
|
||||||
|
"total_posts": row["total_posts"],
|
||||||
|
"posts": row["posts"],
|
||||||
|
"replies": row["replies"],
|
||||||
|
"boosts": row["boosts"],
|
||||||
|
"reactions": row["reactions"],
|
||||||
|
"group_messages": row["group_messages"],
|
||||||
|
"polls": row["polls"],
|
||||||
|
}
|
||||||
|
|
||||||
|
return years
|
||||||
|
|
||||||
|
def _get_global_stats(self):
|
||||||
|
"""Global counters, independent of the monthly breakdown.
|
||||||
|
|
||||||
|
Bridge virtual feeds (and their profiles, posts and follows) are
|
||||||
|
a connection helper, not real accounts, so they never count.
|
||||||
|
"""
|
||||||
|
real_posts = Post.objects.exclude(bridge_urls_q("profile__feed"))
|
||||||
|
return {
|
||||||
|
"registered_feeds": Feed.objects.exclude(bridge_urls_q("url")).count(),
|
||||||
|
"total_accounts": Profile.objects.exclude(bridge_urls_q("feed")).count(),
|
||||||
|
"total_posts": real_posts.count(),
|
||||||
|
"total_follows": Follow.objects.exclude(
|
||||||
|
bridge_urls_q("followed__feed")
|
||||||
|
).count(),
|
||||||
|
"active_groups": real_posts.exclude(group="")
|
||||||
|
.values("group")
|
||||||
|
.distinct()
|
||||||
|
.count(),
|
||||||
|
}
|
||||||
|
|
@ -26,10 +26,11 @@ services:
|
||||||
- .:/app
|
- .:/app
|
||||||
command: ./entrypoint.sh
|
command: ./entrypoint.sh
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD", "python", "manage.py", "check"]
|
test: ["CMD", "python3", "-c", "import http.client; c=http.client.HTTPConnection('localhost',8000,timeout=5); c.request('GET','/'); exit(0 if c.getresponse().status<400 else 1)"]
|
||||||
interval: 30s
|
interval: 2m
|
||||||
timeout: 10s
|
timeout: 10s
|
||||||
retries: 3
|
retries: 3
|
||||||
|
start_period: 40s
|
||||||
|
|
||||||
huey:
|
huey:
|
||||||
build: .
|
build: .
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,6 @@ https://docs.djangoproject.com/en/5.2/ref/settings/
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import os
|
import os
|
||||||
import socket
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
# Build paths inside the project like this: BASE_DIR / 'subdir'.
|
# Build paths inside the project like this: BASE_DIR / 'subdir'.
|
||||||
|
|
@ -69,6 +68,7 @@ INSTALLED_APPS = [
|
||||||
"huey.contrib.djhuey",
|
"huey.contrib.djhuey",
|
||||||
"app.public.apps.PublicConfig",
|
"app.public.apps.PublicConfig",
|
||||||
"app.feeds.apps.FeedsConfig",
|
"app.feeds.apps.FeedsConfig",
|
||||||
|
"app.feedcontent.apps.FeedcontentConfig",
|
||||||
"app.polls.apps.PollsConfig",
|
"app.polls.apps.PollsConfig",
|
||||||
"app.replies.apps.RepliesConfig",
|
"app.replies.apps.RepliesConfig",
|
||||||
"app.mentions.apps.MentionsConfig",
|
"app.mentions.apps.MentionsConfig",
|
||||||
|
|
@ -77,10 +77,18 @@ INSTALLED_APPS = [
|
||||||
"app.notifications.apps.NotificationsConfig",
|
"app.notifications.apps.NotificationsConfig",
|
||||||
"app.search.apps.SearchConfig",
|
"app.search.apps.SearchConfig",
|
||||||
"app.groups.apps.GroupsConfig",
|
"app.groups.apps.GroupsConfig",
|
||||||
|
"app.rss.apps.RssConfig",
|
||||||
|
"app.boosts.apps.BoostsConfig",
|
||||||
|
"app.interactions.apps.InteractionsConfig",
|
||||||
|
"app.sse_notifications.apps.SseNotificationsConfig",
|
||||||
|
"app.profile.apps.ProfileConfig",
|
||||||
|
"app.stats.apps.StatsConfig",
|
||||||
|
"app.bridge.apps.BridgeConfig",
|
||||||
]
|
]
|
||||||
|
|
||||||
MIDDLEWARE = [
|
MIDDLEWARE = [
|
||||||
"django.middleware.security.SecurityMiddleware",
|
"django.middleware.security.SecurityMiddleware",
|
||||||
|
"app.feeds.cors_middleware.CORSMiddleware", # Add CORS headers to all responses
|
||||||
"django.middleware.common.CommonMiddleware",
|
"django.middleware.common.CommonMiddleware",
|
||||||
"django.middleware.csrf.CsrfViewMiddleware",
|
"django.middleware.csrf.CsrfViewMiddleware",
|
||||||
"app.feeds.middleware.RelayMetadataMiddleware", # Add global ETag/Last-Modified headers
|
"app.feeds.middleware.RelayMetadataMiddleware", # Add global ETag/Last-Modified headers
|
||||||
|
|
@ -146,23 +154,12 @@ REST_FRAMEWORK = {
|
||||||
}
|
}
|
||||||
|
|
||||||
# Cache configuration
|
# Cache configuration
|
||||||
|
# In DEBUG (dev/test) we use DummyCache so the suite runs without Redis.
|
||||||
|
# In production we always use Redis: a startup probe is unsafe because if Redis
|
||||||
|
# is still warming up at import time the process would silently fall back to
|
||||||
|
# DummyCache for its entire lifetime, breaking cache invalidation.
|
||||||
|
|
||||||
|
if DEBUG:
|
||||||
def redis_available():
|
|
||||||
"""Check if Redis is available"""
|
|
||||||
try:
|
|
||||||
redis_host = os.environ.get("REDIS_HOST", "redis")
|
|
||||||
redis_port = int(os.environ.get("REDIS_PORT", 6379))
|
|
||||||
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
|
||||||
sock.settimeout(1)
|
|
||||||
result = sock.connect_ex((redis_host, redis_port))
|
|
||||||
sock.close()
|
|
||||||
return result == 0
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
if DEBUG or not redis_available():
|
|
||||||
CACHES = {
|
CACHES = {
|
||||||
"default": {
|
"default": {
|
||||||
"BACKEND": "django.core.cache.backends.dummy.DummyCache",
|
"BACKEND": "django.core.cache.backends.dummy.DummyCache",
|
||||||
|
|
|
||||||
|
|
@ -22,12 +22,20 @@ from app.feeds.views import FeedsView
|
||||||
urlpatterns = [
|
urlpatterns = [
|
||||||
path("", root_view, name="root"),
|
path("", root_view, name="root"),
|
||||||
path("feeds/", FeedsView.as_view(), name="feeds"),
|
path("feeds/", FeedsView.as_view(), name="feeds"),
|
||||||
|
path("feed-content/", include("app.feedcontent.urls")),
|
||||||
path("mentions/", include("app.mentions.urls")),
|
path("mentions/", include("app.mentions.urls")),
|
||||||
path("reactions/", include("app.reactions.urls")),
|
path("reactions/", include("app.reactions.urls")),
|
||||||
path("replies-to/", include("app.repliesto.urls")),
|
path("replies-to/", include("app.repliesto.urls")),
|
||||||
|
path("boosts/", include("app.boosts.urls")),
|
||||||
|
path("interactions/", include("app.interactions.urls")),
|
||||||
path("notifications/", include("app.notifications.urls")),
|
path("notifications/", include("app.notifications.urls")),
|
||||||
path("replies/", include("app.replies.urls")),
|
path("replies/", include("app.replies.urls")),
|
||||||
path("search/", include("app.search.urls")),
|
path("search/", include("app.search.urls")),
|
||||||
path("groups/", include("app.groups.urls")),
|
path("groups/", include("app.groups.urls")),
|
||||||
path("polls/", include("app.polls.urls")),
|
path("polls/", include("app.polls.urls")),
|
||||||
|
path("rss.xml", include("app.rss.urls")),
|
||||||
|
path("sse/", include("app.sse_notifications.urls")),
|
||||||
|
path("profile/", include("app.profile.urls")),
|
||||||
|
path("stats/", include("app.stats.urls")),
|
||||||
|
path("bridge/", include("app.bridge.urls")),
|
||||||
]
|
]
|
||||||
|
|
|
||||||
20
diagram.mmd
20
diagram.mmd
|
|
@ -1,8 +1,7 @@
|
||||||
graph TD
|
graph TD
|
||||||
List["📋 List nodes"]
|
Node1["🖥️ Relay 1"]
|
||||||
Node1["🖥️ Node 1"]
|
Node2["🖥️ Relay 2"]
|
||||||
Node2["🖥️ Node 2"]
|
Node3["🖥️ Relay 3"]
|
||||||
Node3["🖥️ Node 3"]
|
|
||||||
|
|
||||||
%% Social.org instances with icons
|
%% Social.org instances with icons
|
||||||
Social1_1["📄 social.org"]
|
Social1_1["📄 social.org"]
|
||||||
|
|
@ -12,11 +11,6 @@ graph TD
|
||||||
Social3_1["📄 social.org"]
|
Social3_1["📄 social.org"]
|
||||||
Social3_2["📄 social.org"]
|
Social3_2["📄 social.org"]
|
||||||
|
|
||||||
%% Parent-child connections with labels
|
|
||||||
List -.->|"Get"| Node1
|
|
||||||
List -.->|"Get"| Node2
|
|
||||||
List -.->|"Get"| Node3
|
|
||||||
|
|
||||||
%% Node to social.org connections
|
%% Node to social.org connections
|
||||||
Social1_1 -->|"⚓ Connects"| Node1
|
Social1_1 -->|"⚓ Connects"| Node1
|
||||||
Social1_2 -->|"⚓ Connects"| Node1
|
Social1_2 -->|"⚓ Connects"| Node1
|
||||||
|
|
@ -26,16 +20,14 @@ graph TD
|
||||||
Social3_2 -->|"⚓ Connects"| Node3
|
Social3_2 -->|"⚓ Connects"| Node3
|
||||||
|
|
||||||
%% Bidirectional connections between nodes
|
%% Bidirectional connections between nodes
|
||||||
Node1 <-.->|"👥 Share Users"| Node2
|
Node1 <-.->|"👥 Share Feeds"| Node2
|
||||||
Node2 <-.->|"👥 Share Users"| Node3
|
Node2 <-.->|"👥 Share Feeds"| Node3
|
||||||
Node1 <-.->|"👥 Share Users"| Node3
|
Node1 <-.->|"👥 Share Feeds"| Node3
|
||||||
|
|
||||||
%% Modern color scheme with gradients
|
%% Modern color scheme with gradients
|
||||||
classDef socialStyle fill:#667eea,stroke:#764ba2,stroke-width:3px,color:#fff,font-weight:bold
|
classDef socialStyle fill:#667eea,stroke:#764ba2,stroke-width:3px,color:#fff,font-weight:bold
|
||||||
classDef nodeStyle fill:#f093fb,stroke:#f5576c,stroke-width:3px,color:#fff,font-weight:bold
|
classDef nodeStyle fill:#f093fb,stroke:#f5576c,stroke-width:3px,color:#fff,font-weight:bold
|
||||||
classDef listStyle fill:#4facfe,stroke:#00f2fe,stroke-width:4px,color:#fff,font-weight:bold
|
|
||||||
|
|
||||||
%% Apply styles
|
%% Apply styles
|
||||||
class Social1_1,Social1_2,Social2_1,Social2_2,Social3_1,Social3_2 socialStyle
|
class Social1_1,Social1_2,Social2_1,Social2_2,Social3_1,Social3_2 socialStyle
|
||||||
class Node1,Node2,Node3 nodeStyle
|
class Node1,Node2,Node3 nodeStyle
|
||||||
class List listStyle
|
|
||||||
|
|
|
||||||
|
|
@ -11,6 +11,6 @@ python manage.py migrate
|
||||||
echo "🔍 Checking Django configuration..."
|
echo "🔍 Checking Django configuration..."
|
||||||
python manage.py check
|
python manage.py check
|
||||||
|
|
||||||
# Start Django development server
|
# Start application server
|
||||||
echo "🎯 Starting Django development server on 0.0.0.0:8000..."
|
echo "🎯 Starting uvicorn on 0.0.0.0:8000..."
|
||||||
exec python manage.py runserver 0.0.0.0:8000
|
exec uvicorn core.asgi:application --host 0.0.0.0 --port 8000 --workers 4
|
||||||
|
|
@ -27,6 +27,15 @@ http {
|
||||||
proxy_send_timeout 60s;
|
proxy_send_timeout 60s;
|
||||||
proxy_read_timeout 60s;
|
proxy_read_timeout 60s;
|
||||||
|
|
||||||
|
# SSE location - disable buffering for streaming responses
|
||||||
|
location /sse/ {
|
||||||
|
proxy_pass http://django_app;
|
||||||
|
proxy_buffering off;
|
||||||
|
proxy_cache off;
|
||||||
|
proxy_read_timeout 3600s;
|
||||||
|
gzip off;
|
||||||
|
}
|
||||||
|
|
||||||
# Main location
|
# Main location
|
||||||
location / {
|
location / {
|
||||||
proxy_pass http://django_app;
|
proxy_pass http://django_app;
|
||||||
|
|
|
||||||
|
|
@ -11,6 +11,8 @@ dependencies = [
|
||||||
"huey>=2.5.0",
|
"huey>=2.5.0",
|
||||||
"redis>=4.0.0",
|
"redis>=4.0.0",
|
||||||
"requests>=2.31.0",
|
"requests>=2.31.0",
|
||||||
|
"python-dateutil>=2.8.0",
|
||||||
|
"feedparser>=6.0.0",
|
||||||
"pytest>=7.0.0",
|
"pytest>=7.0.0",
|
||||||
"pytest-django>=4.5.0",
|
"pytest-django>=4.5.0",
|
||||||
]
|
]
|
||||||
|
|
|
||||||
2
relay-list.txt
Normal file
2
relay-list.txt
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
https://relay.org-social.org
|
||||||
|
https://orgs-relay.adsan.dev
|
||||||
|
|
@ -6,5 +6,8 @@ redis>=4.0.0
|
||||||
django-redis>=5.0.0
|
django-redis>=5.0.0
|
||||||
requests>=2.31.0
|
requests>=2.31.0
|
||||||
python-dateutil>=2.8.0
|
python-dateutil>=2.8.0
|
||||||
|
feedparser>=6.0.0
|
||||||
pytest>=7.0.0
|
pytest>=7.0.0
|
||||||
pytest-django>=4.5.0
|
pytest-django>=4.5.0
|
||||||
|
org-python>=0.3.1
|
||||||
|
uvicorn>=0.30.0
|
||||||
Loading…
Add table
Reference in a new issue