Files
outline/server/queues/tasks/DeliverWebhookTask.ts
Tom Moor 10f86ed218 feat: Webhooks (#3691)
* Webhooks (#3607)

* Get the migration and the model setup. Also make the sample env file a bit easier to use. Now just requires setting a SECRET_KEY and besides that will boot up from the sample

* WIP: Start getting a Webhook page created. Just the skeleton state right now

* WIP: Getting a form created to create webhooks, need to bring in react-hook-forms now

* WIP: Get library installed and make TS happy

* Get a few checkboxes ready to go

* Get creating and destroying working with a decent start to a frontend

* Didn't mean to enable this

* Remove eslint and fix other random typescript issue

* Rename some events to be more realistic

* Revert these changes

* PR review comments around policies. Also make sure this inherits from IdModel so it actually gets an id

* Allow any admin on the team to edit webhooks

* Start sending some webhooks for some User events

* Make sure the URL is valid

* Start recording webhook deliveries

* Make sure to verify if the subscription is for the type of event we are looking at

* Refactor sending Webhooks and follow better webhook schema

This creates a presenter to unify the format of webhooks. We also
extract the sending of webhooks and recording their deliveries to a
method than can be used by each of the different event type methods

We also add a status to WebhookDelivery since we need to save the record
before we make the HTTP request to get its id. Then once we make the
request and get a response we can update the delivery with the HTTP info

* Turn off a subscription that has failed for the last 25 deliveries

* Get a first spec passing. Found a bug in my returning of promises so good to patch that up now

* This looks nicer

* Get some tests added for the processor

* Add cron task to delete older webhooks

* Add Document Events to the Processor

* Revisions, FileOperations and Collections

* Get all the server side events added to the processor and make Typescript make sure they are all accounted for

* Get all the events added to the Frontend and work on styling them a bit, still needs some love though

* Get UI styled up a bit

* Get events wired up for webhook subscriptions

* Get delete events working and test at least one variant of them

* Get deletes working and actually make sure to send the model id in the webhook

* Remove webhook secrets from this slice

* Add disabled label for subscriptions that are disabled

* Make sure to cascade the delete

* Reorg this file a bit

* Fix association

* I removed secret for the moment

* Apply Copy changes from PR Review

Co-authored-by: Tom Moor <tom.moor@gmail.com>

* Actually apply the copy changes

TIL that if you Resolve a conversation it _also_ removes the 'staged suggestion' from your list on Github

Co-authored-by: Tom Moor <tom.moor@gmail.com>

* Update app/scenes/Settings/Webhooks.tsx

Missed this copy change before

Co-authored-by: Tom Moor <tom.moor@gmail.com>

* Add disabled as yellow badge

* Resolve frontend comments

* Fixup Schema a bit and remove the dependency on the subscription

* Add test to make sure we don't disable until there are enough failures, and fix code to actually do that. Also some test fixes from the json response shape changes

* Fix WebhookDeliveries to store the responses as Text instead of blobs

* Switch to text better for response bodies, this is using the helpers better and makes the code read better

* Move the logic to a task but run in through the processor cause the tests expect that right now, moving the tests over next

* Split up the tests and actually enqueue the events from the WebhookProcessor instead of doing them inline

* Allow any team admin to see any webhook subscription for the team

* Add the indexes based on our lookup patterns

* Run eslint --fix to fix auto correct issues from when I tried to use Github to merge copy changes

* Allow subscriptions to be edited after creation

* Types caught that I didn't add the new event to the webhook processor, also added it to the frontend here

* I think this will get these into the translations file

* Catch a few more translations, use styled components better and remove usage of webhook subscription in the copy

Co-authored-by: Tom Moor <tom.moor@gmail.com>

* fix: tsc
fix: Document model payload empty

* fix: Revision webhook payload
Add custom UA for hooks

* Add webhooks icon, move under Integrations settings
Some spacing fixes

* Add actorId to webhook payloads

* Add View and ApiKey event types

* Spacing tweaks, fix team payload

* fix: Webhook not disabled after 25 failures

* fix: Enable webhook when editing if previously disabled

* fix: Correctly store response headers

* fix: Error in json/parsing/presentation results in hanging 'pending' webhook delivery

* fix: Awkward payload for users.invite webhook

* Add BaseEvent, ShareEvent

* fix: Add share events to form

* fix: Move webhook delivery cleanup to single DB call
Remove some unused abstraction

* Add user, collection, group context to membership webhook events
Some associated refactoring

Co-authored-by: Corey Alexander <coreyja@gmail.com>
2022-06-28 22:44:50 -07:00

561 lines
14 KiB
TypeScript

import invariant from "invariant";
import env from "@server/env";
import Logger from "@server/logging/Logger";
import {
Collection,
FileOperation,
Group,
Integration,
Pin,
Star,
Team,
WebhookDelivery,
WebhookSubscription,
Document,
User,
Revision,
View,
Share,
CollectionUser,
CollectionGroup,
GroupUser,
} from "@server/models";
import {
presentCollection,
presentDocument,
presentRevision,
presentFileOperation,
presentGroup,
presentIntegration,
presentPin,
presentStar,
presentTeam,
presentUser,
presentWebhook,
presentWebhookSubscription,
presentView,
presentShare,
presentMembership,
presentGroupMembership,
presentCollectionGroupMembership,
} from "@server/presenters";
import { WebhookPayload } from "@server/presenters/webhook";
import {
CollectionEvent,
CollectionGroupEvent,
CollectionUserEvent,
DocumentEvent,
Event,
FileOperationEvent,
GroupEvent,
GroupUserEvent,
IntegrationEvent,
PinEvent,
RevisionEvent,
ShareEvent,
StarEvent,
TeamEvent,
UserEvent,
ViewEvent,
WebhookSubscriptionEvent,
} from "@server/types";
import BaseTask from "./BaseTask";
function assertUnreachable(event: never) {
Logger.warn(`DeliverWebhookTask did not handle ${(event as any).name}`);
}
type Props = {
subscriptionId: string;
event: Event;
};
export default class DeliverWebhookTask extends BaseTask<Props> {
public async perform({ subscriptionId, event }: Props) {
const subscription = await WebhookSubscription.findByPk(subscriptionId);
invariant(subscription, "Subscription not found");
Logger.info(
"task",
`DeliverWebhookTask: ${event.name} for ${subscription.name}`
);
switch (event.name) {
case "api_keys.create":
case "api_keys.delete":
// Ignored
return;
case "users.create":
case "users.signin":
case "users.signout":
case "users.update":
case "users.suspend":
case "users.activate":
case "users.delete":
case "users.invite":
await this.handleUserEvent(subscription, event);
return;
case "documents.create":
case "documents.publish":
case "documents.unpublish":
case "documents.delete":
case "documents.permanent_delete":
case "documents.archive":
case "documents.unarchive":
case "documents.restore":
case "documents.star":
case "documents.unstar":
case "documents.move":
case "documents.update":
case "documents.title_change":
await this.handleDocumentEvent(subscription, event);
return;
case "documents.update.delayed":
case "documents.update.debounced":
// Ignored
return;
case "revisions.create":
await this.handleRevisionEvent(subscription, event);
return;
case "fileOperations.create":
case "fileOperations.update":
case "fileOperation.delete":
await this.handleFileOperationEvent(subscription, event);
return;
case "collections.create":
case "collections.update":
case "collections.delete":
case "collections.move":
case "collections.permission_changed":
await this.handleCollectionEvent(subscription, event);
return;
case "collections.add_user":
case "collections.remove_user":
await this.handleCollectionUserEvent(subscription, event);
return;
case "collections.add_group":
case "collections.remove_group":
await this.handleCollectionGroupEvent(subscription, event);
return;
case "groups.create":
case "groups.update":
case "groups.delete":
await this.handleGroupEvent(subscription, event);
return;
case "groups.add_user":
case "groups.remove_user":
await this.handleGroupUserEvent(subscription, event);
return;
case "integrations.create":
case "integrations.update":
await this.handleIntegrationEvent(subscription, event);
return;
case "teams.update":
await this.handleTeamEvent(subscription, event);
return;
case "pins.create":
case "pins.update":
case "pins.delete":
await this.handlePinEvent(subscription, event);
return;
case "stars.create":
case "stars.update":
case "stars.delete":
await this.handleStarEvent(subscription, event);
return;
case "shares.create":
case "shares.update":
case "shares.revoke":
await this.handleShareEvent(subscription, event);
return;
case "webhook_subscriptions.create":
case "webhook_subscriptions.delete":
case "webhook_subscriptions.update":
await this.handleWebhookSubscriptionEvent(subscription, event);
return;
case "views.create":
await this.handleViewEvent(subscription, event);
return;
default:
assertUnreachable(event);
}
}
private async handleWebhookSubscriptionEvent(
subscription: WebhookSubscription,
event: WebhookSubscriptionEvent
): Promise<void> {
const model = await WebhookSubscription.findByPk(event.modelId, {
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: event.modelId,
model: model && presentWebhookSubscription(model),
},
});
}
private async handleViewEvent(
subscription: WebhookSubscription,
event: ViewEvent
): Promise<void> {
const model = await View.scope("withUser").findByPk(event.modelId, {
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: event.modelId,
model: model && presentView(model),
},
});
}
private async handleStarEvent(
subscription: WebhookSubscription,
event: StarEvent
): Promise<void> {
const model = await Star.findByPk(event.modelId, {
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: event.modelId,
model: model && presentStar(model),
},
});
}
private async handleShareEvent(
subscription: WebhookSubscription,
event: ShareEvent
): Promise<void> {
const model = await Share.findByPk(event.modelId, {
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: event.modelId,
model: model && presentShare(model),
},
});
}
private async handlePinEvent(
subscription: WebhookSubscription,
event: PinEvent
): Promise<void> {
const model = await Pin.findByPk(event.modelId, {
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: event.modelId,
model: model && presentPin(model),
},
});
}
private async handleTeamEvent(
subscription: WebhookSubscription,
event: TeamEvent
): Promise<void> {
const model = await Team.scope("withDomains").findByPk(event.teamId, {
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: event.teamId,
model: model && presentTeam(model),
},
});
}
private async handleIntegrationEvent(
subscription: WebhookSubscription,
event: IntegrationEvent
): Promise<void> {
const model = await Integration.findByPk(event.modelId, {
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: event.modelId,
model: model && presentIntegration(model),
},
});
}
private async handleGroupEvent(
subscription: WebhookSubscription,
event: GroupEvent
): Promise<void> {
const model = await Group.findByPk(event.modelId, {
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: event.modelId,
model: model && presentGroup(model),
},
});
}
private async handleGroupUserEvent(
subscription: WebhookSubscription,
event: GroupUserEvent
): Promise<void> {
const model = await GroupUser.scope(["withUser", "withGroup"]).findOne({
where: {
groupId: event.modelId,
userId: event.userId,
},
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: `${event.userId}-${event.modelId}`,
model: model && presentGroupMembership(model),
group: model && presentGroup(model.group),
user: model && presentUser(model.user),
},
});
}
private async handleCollectionEvent(
subscription: WebhookSubscription,
event: CollectionEvent
): Promise<void> {
const model = await Collection.findByPk(event.collectionId, {
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: event.collectionId,
model: model && presentCollection(model),
},
});
}
private async handleCollectionUserEvent(
subscription: WebhookSubscription,
event: CollectionUserEvent
): Promise<void> {
const model = await CollectionUser.scope([
"withUser",
"withCollection",
]).findOne({
where: {
collectionId: event.collectionId,
userId: event.userId,
},
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: `${event.userId}-${event.collectionId}`,
model: model && presentMembership(model),
collection: model && presentCollection(model.collection),
user: model && presentUser(model.user),
},
});
}
private async handleCollectionGroupEvent(
subscription: WebhookSubscription,
event: CollectionGroupEvent
): Promise<void> {
const model = await CollectionGroup.scope([
"withGroup",
"withCollection",
]).findOne({
where: {
collectionId: event.collectionId,
groupId: event.modelId,
},
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: `${event.modelId}-${event.collectionId}`,
model: model && presentCollectionGroupMembership(model),
collection: model && presentCollection(model.collection),
group: model && presentGroup(model.group),
},
});
}
private async handleFileOperationEvent(
subscription: WebhookSubscription,
event: FileOperationEvent
): Promise<void> {
const model = await FileOperation.findByPk(event.modelId, {
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: event.modelId,
model: model && presentFileOperation(model),
},
});
}
private async handleDocumentEvent(
subscription: WebhookSubscription,
event: DocumentEvent
): Promise<void> {
const model = await Document.findByPk(event.documentId, {
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: event.documentId,
model: model && (await presentDocument(model)),
},
});
}
private async handleRevisionEvent(
subscription: WebhookSubscription,
event: RevisionEvent
): Promise<void> {
const model = await Revision.findByPk(event.modelId, {
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: event.modelId,
model: model && (await presentRevision(model)),
},
});
}
private async handleUserEvent(
subscription: WebhookSubscription,
event: UserEvent
): Promise<void> {
const model = await User.findByPk(event.userId, {
paranoid: false,
});
await this.sendWebhook({
event,
subscription,
payload: {
id: event.userId,
model: model && presentUser(model),
},
});
}
private async sendWebhook({
event,
subscription,
payload,
}: {
event: Event;
subscription: WebhookSubscription;
payload: WebhookPayload;
}) {
const delivery = await WebhookDelivery.create({
webhookSubscriptionId: subscription.id,
status: "pending",
});
let response, requestBody, requestHeaders, status;
try {
requestBody = presentWebhook({
event,
delivery,
payload,
});
requestHeaders = {
"Content-Type": "application/json",
"user-agent": `Outline-Webhooks${env.VERSION ? `/${env.VERSION}` : ""}`,
};
response = await fetch(subscription.url, {
method: "POST",
headers: requestHeaders,
body: JSON.stringify(requestBody),
});
status = response.ok ? "success" : "failed";
} catch (err) {
status = "failed";
}
await delivery.update({
status,
statusCode: response ? response.status : null,
requestBody,
requestHeaders,
responseBody: response ? await response.text() : "",
responseHeaders: response
? Object.fromEntries(response.headers.entries())
: {},
});
if (response && !response.ok) {
const recentDeliveries = await WebhookDelivery.findAll({
where: {
webhookSubscriptionId: subscription.id,
},
order: [["createdAt", "DESC"]],
limit: 25,
});
const allFailed = recentDeliveries.every(
(delivery) => delivery.status === "failed"
);
if (recentDeliveries.length === 25 && allFailed) {
await subscription.update({ enabled: false });
}
}
}
}