* Webhooks (#3607) * Get the migration and the model setup. Also make the sample env file a bit easier to use. Now just requires setting a SECRET_KEY and besides that will boot up from the sample * WIP: Start getting a Webhook page created. Just the skeleton state right now * WIP: Getting a form created to create webhooks, need to bring in react-hook-forms now * WIP: Get library installed and make TS happy * Get a few checkboxes ready to go * Get creating and destroying working with a decent start to a frontend * Didn't mean to enable this * Remove eslint and fix other random typescript issue * Rename some events to be more realistic * Revert these changes * PR review comments around policies. Also make sure this inherits from IdModel so it actually gets an id * Allow any admin on the team to edit webhooks * Start sending some webhooks for some User events * Make sure the URL is valid * Start recording webhook deliveries * Make sure to verify if the subscription is for the type of event we are looking at * Refactor sending Webhooks and follow better webhook schema This creates a presenter to unify the format of webhooks. We also extract the sending of webhooks and recording their deliveries to a method than can be used by each of the different event type methods We also add a status to WebhookDelivery since we need to save the record before we make the HTTP request to get its id. Then once we make the request and get a response we can update the delivery with the HTTP info * Turn off a subscription that has failed for the last 25 deliveries * Get a first spec passing. Found a bug in my returning of promises so good to patch that up now * This looks nicer * Get some tests added for the processor * Add cron task to delete older webhooks * Add Document Events to the Processor * Revisions, FileOperations and Collections * Get all the server side events added to the processor and make Typescript make sure they are all accounted for * Get all the events added to the Frontend and work on styling them a bit, still needs some love though * Get UI styled up a bit * Get events wired up for webhook subscriptions * Get delete events working and test at least one variant of them * Get deletes working and actually make sure to send the model id in the webhook * Remove webhook secrets from this slice * Add disabled label for subscriptions that are disabled * Make sure to cascade the delete * Reorg this file a bit * Fix association * I removed secret for the moment * Apply Copy changes from PR Review Co-authored-by: Tom Moor <tom.moor@gmail.com> * Actually apply the copy changes TIL that if you Resolve a conversation it _also_ removes the 'staged suggestion' from your list on Github Co-authored-by: Tom Moor <tom.moor@gmail.com> * Update app/scenes/Settings/Webhooks.tsx Missed this copy change before Co-authored-by: Tom Moor <tom.moor@gmail.com> * Add disabled as yellow badge * Resolve frontend comments * Fixup Schema a bit and remove the dependency on the subscription * Add test to make sure we don't disable until there are enough failures, and fix code to actually do that. Also some test fixes from the json response shape changes * Fix WebhookDeliveries to store the responses as Text instead of blobs * Switch to text better for response bodies, this is using the helpers better and makes the code read better * Move the logic to a task but run in through the processor cause the tests expect that right now, moving the tests over next * Split up the tests and actually enqueue the events from the WebhookProcessor instead of doing them inline * Allow any team admin to see any webhook subscription for the team * Add the indexes based on our lookup patterns * Run eslint --fix to fix auto correct issues from when I tried to use Github to merge copy changes * Allow subscriptions to be edited after creation * Types caught that I didn't add the new event to the webhook processor, also added it to the frontend here * I think this will get these into the translations file * Catch a few more translations, use styled components better and remove usage of webhook subscription in the copy Co-authored-by: Tom Moor <tom.moor@gmail.com> * fix: tsc fix: Document model payload empty * fix: Revision webhook payload Add custom UA for hooks * Add webhooks icon, move under Integrations settings Some spacing fixes * Add actorId to webhook payloads * Add View and ApiKey event types * Spacing tweaks, fix team payload * fix: Webhook not disabled after 25 failures * fix: Enable webhook when editing if previously disabled * fix: Correctly store response headers * fix: Error in json/parsing/presentation results in hanging 'pending' webhook delivery * fix: Awkward payload for users.invite webhook * Add BaseEvent, ShareEvent * fix: Add share events to form * fix: Move webhook delivery cleanup to single DB call Remove some unused abstraction * Add user, collection, group context to membership webhook events Some associated refactoring Co-authored-by: Corey Alexander <coreyja@gmail.com>
561 lines
14 KiB
TypeScript
561 lines
14 KiB
TypeScript
import invariant from "invariant";
|
|
import env from "@server/env";
|
|
import Logger from "@server/logging/Logger";
|
|
import {
|
|
Collection,
|
|
FileOperation,
|
|
Group,
|
|
Integration,
|
|
Pin,
|
|
Star,
|
|
Team,
|
|
WebhookDelivery,
|
|
WebhookSubscription,
|
|
Document,
|
|
User,
|
|
Revision,
|
|
View,
|
|
Share,
|
|
CollectionUser,
|
|
CollectionGroup,
|
|
GroupUser,
|
|
} from "@server/models";
|
|
import {
|
|
presentCollection,
|
|
presentDocument,
|
|
presentRevision,
|
|
presentFileOperation,
|
|
presentGroup,
|
|
presentIntegration,
|
|
presentPin,
|
|
presentStar,
|
|
presentTeam,
|
|
presentUser,
|
|
presentWebhook,
|
|
presentWebhookSubscription,
|
|
presentView,
|
|
presentShare,
|
|
presentMembership,
|
|
presentGroupMembership,
|
|
presentCollectionGroupMembership,
|
|
} from "@server/presenters";
|
|
import { WebhookPayload } from "@server/presenters/webhook";
|
|
import {
|
|
CollectionEvent,
|
|
CollectionGroupEvent,
|
|
CollectionUserEvent,
|
|
DocumentEvent,
|
|
Event,
|
|
FileOperationEvent,
|
|
GroupEvent,
|
|
GroupUserEvent,
|
|
IntegrationEvent,
|
|
PinEvent,
|
|
RevisionEvent,
|
|
ShareEvent,
|
|
StarEvent,
|
|
TeamEvent,
|
|
UserEvent,
|
|
ViewEvent,
|
|
WebhookSubscriptionEvent,
|
|
} from "@server/types";
|
|
import BaseTask from "./BaseTask";
|
|
|
|
function assertUnreachable(event: never) {
|
|
Logger.warn(`DeliverWebhookTask did not handle ${(event as any).name}`);
|
|
}
|
|
|
|
type Props = {
|
|
subscriptionId: string;
|
|
event: Event;
|
|
};
|
|
|
|
export default class DeliverWebhookTask extends BaseTask<Props> {
|
|
public async perform({ subscriptionId, event }: Props) {
|
|
const subscription = await WebhookSubscription.findByPk(subscriptionId);
|
|
invariant(subscription, "Subscription not found");
|
|
|
|
Logger.info(
|
|
"task",
|
|
`DeliverWebhookTask: ${event.name} for ${subscription.name}`
|
|
);
|
|
|
|
switch (event.name) {
|
|
case "api_keys.create":
|
|
case "api_keys.delete":
|
|
// Ignored
|
|
return;
|
|
case "users.create":
|
|
case "users.signin":
|
|
case "users.signout":
|
|
case "users.update":
|
|
case "users.suspend":
|
|
case "users.activate":
|
|
case "users.delete":
|
|
case "users.invite":
|
|
await this.handleUserEvent(subscription, event);
|
|
return;
|
|
case "documents.create":
|
|
case "documents.publish":
|
|
case "documents.unpublish":
|
|
case "documents.delete":
|
|
case "documents.permanent_delete":
|
|
case "documents.archive":
|
|
case "documents.unarchive":
|
|
case "documents.restore":
|
|
case "documents.star":
|
|
case "documents.unstar":
|
|
case "documents.move":
|
|
case "documents.update":
|
|
case "documents.title_change":
|
|
await this.handleDocumentEvent(subscription, event);
|
|
return;
|
|
case "documents.update.delayed":
|
|
case "documents.update.debounced":
|
|
// Ignored
|
|
return;
|
|
case "revisions.create":
|
|
await this.handleRevisionEvent(subscription, event);
|
|
return;
|
|
case "fileOperations.create":
|
|
case "fileOperations.update":
|
|
case "fileOperation.delete":
|
|
await this.handleFileOperationEvent(subscription, event);
|
|
return;
|
|
case "collections.create":
|
|
case "collections.update":
|
|
case "collections.delete":
|
|
case "collections.move":
|
|
case "collections.permission_changed":
|
|
await this.handleCollectionEvent(subscription, event);
|
|
return;
|
|
case "collections.add_user":
|
|
case "collections.remove_user":
|
|
await this.handleCollectionUserEvent(subscription, event);
|
|
return;
|
|
case "collections.add_group":
|
|
case "collections.remove_group":
|
|
await this.handleCollectionGroupEvent(subscription, event);
|
|
return;
|
|
case "groups.create":
|
|
case "groups.update":
|
|
case "groups.delete":
|
|
await this.handleGroupEvent(subscription, event);
|
|
return;
|
|
case "groups.add_user":
|
|
case "groups.remove_user":
|
|
await this.handleGroupUserEvent(subscription, event);
|
|
return;
|
|
case "integrations.create":
|
|
case "integrations.update":
|
|
await this.handleIntegrationEvent(subscription, event);
|
|
return;
|
|
case "teams.update":
|
|
await this.handleTeamEvent(subscription, event);
|
|
return;
|
|
case "pins.create":
|
|
case "pins.update":
|
|
case "pins.delete":
|
|
await this.handlePinEvent(subscription, event);
|
|
return;
|
|
case "stars.create":
|
|
case "stars.update":
|
|
case "stars.delete":
|
|
await this.handleStarEvent(subscription, event);
|
|
return;
|
|
case "shares.create":
|
|
case "shares.update":
|
|
case "shares.revoke":
|
|
await this.handleShareEvent(subscription, event);
|
|
return;
|
|
case "webhook_subscriptions.create":
|
|
case "webhook_subscriptions.delete":
|
|
case "webhook_subscriptions.update":
|
|
await this.handleWebhookSubscriptionEvent(subscription, event);
|
|
return;
|
|
case "views.create":
|
|
await this.handleViewEvent(subscription, event);
|
|
return;
|
|
default:
|
|
assertUnreachable(event);
|
|
}
|
|
}
|
|
|
|
private async handleWebhookSubscriptionEvent(
|
|
subscription: WebhookSubscription,
|
|
event: WebhookSubscriptionEvent
|
|
): Promise<void> {
|
|
const model = await WebhookSubscription.findByPk(event.modelId, {
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: event.modelId,
|
|
model: model && presentWebhookSubscription(model),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async handleViewEvent(
|
|
subscription: WebhookSubscription,
|
|
event: ViewEvent
|
|
): Promise<void> {
|
|
const model = await View.scope("withUser").findByPk(event.modelId, {
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: event.modelId,
|
|
model: model && presentView(model),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async handleStarEvent(
|
|
subscription: WebhookSubscription,
|
|
event: StarEvent
|
|
): Promise<void> {
|
|
const model = await Star.findByPk(event.modelId, {
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: event.modelId,
|
|
model: model && presentStar(model),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async handleShareEvent(
|
|
subscription: WebhookSubscription,
|
|
event: ShareEvent
|
|
): Promise<void> {
|
|
const model = await Share.findByPk(event.modelId, {
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: event.modelId,
|
|
model: model && presentShare(model),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async handlePinEvent(
|
|
subscription: WebhookSubscription,
|
|
event: PinEvent
|
|
): Promise<void> {
|
|
const model = await Pin.findByPk(event.modelId, {
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: event.modelId,
|
|
model: model && presentPin(model),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async handleTeamEvent(
|
|
subscription: WebhookSubscription,
|
|
event: TeamEvent
|
|
): Promise<void> {
|
|
const model = await Team.scope("withDomains").findByPk(event.teamId, {
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: event.teamId,
|
|
model: model && presentTeam(model),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async handleIntegrationEvent(
|
|
subscription: WebhookSubscription,
|
|
event: IntegrationEvent
|
|
): Promise<void> {
|
|
const model = await Integration.findByPk(event.modelId, {
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: event.modelId,
|
|
model: model && presentIntegration(model),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async handleGroupEvent(
|
|
subscription: WebhookSubscription,
|
|
event: GroupEvent
|
|
): Promise<void> {
|
|
const model = await Group.findByPk(event.modelId, {
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: event.modelId,
|
|
model: model && presentGroup(model),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async handleGroupUserEvent(
|
|
subscription: WebhookSubscription,
|
|
event: GroupUserEvent
|
|
): Promise<void> {
|
|
const model = await GroupUser.scope(["withUser", "withGroup"]).findOne({
|
|
where: {
|
|
groupId: event.modelId,
|
|
userId: event.userId,
|
|
},
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: `${event.userId}-${event.modelId}`,
|
|
model: model && presentGroupMembership(model),
|
|
group: model && presentGroup(model.group),
|
|
user: model && presentUser(model.user),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async handleCollectionEvent(
|
|
subscription: WebhookSubscription,
|
|
event: CollectionEvent
|
|
): Promise<void> {
|
|
const model = await Collection.findByPk(event.collectionId, {
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: event.collectionId,
|
|
model: model && presentCollection(model),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async handleCollectionUserEvent(
|
|
subscription: WebhookSubscription,
|
|
event: CollectionUserEvent
|
|
): Promise<void> {
|
|
const model = await CollectionUser.scope([
|
|
"withUser",
|
|
"withCollection",
|
|
]).findOne({
|
|
where: {
|
|
collectionId: event.collectionId,
|
|
userId: event.userId,
|
|
},
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: `${event.userId}-${event.collectionId}`,
|
|
model: model && presentMembership(model),
|
|
collection: model && presentCollection(model.collection),
|
|
user: model && presentUser(model.user),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async handleCollectionGroupEvent(
|
|
subscription: WebhookSubscription,
|
|
event: CollectionGroupEvent
|
|
): Promise<void> {
|
|
const model = await CollectionGroup.scope([
|
|
"withGroup",
|
|
"withCollection",
|
|
]).findOne({
|
|
where: {
|
|
collectionId: event.collectionId,
|
|
groupId: event.modelId,
|
|
},
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: `${event.modelId}-${event.collectionId}`,
|
|
model: model && presentCollectionGroupMembership(model),
|
|
collection: model && presentCollection(model.collection),
|
|
group: model && presentGroup(model.group),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async handleFileOperationEvent(
|
|
subscription: WebhookSubscription,
|
|
event: FileOperationEvent
|
|
): Promise<void> {
|
|
const model = await FileOperation.findByPk(event.modelId, {
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: event.modelId,
|
|
model: model && presentFileOperation(model),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async handleDocumentEvent(
|
|
subscription: WebhookSubscription,
|
|
event: DocumentEvent
|
|
): Promise<void> {
|
|
const model = await Document.findByPk(event.documentId, {
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: event.documentId,
|
|
model: model && (await presentDocument(model)),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async handleRevisionEvent(
|
|
subscription: WebhookSubscription,
|
|
event: RevisionEvent
|
|
): Promise<void> {
|
|
const model = await Revision.findByPk(event.modelId, {
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: event.modelId,
|
|
model: model && (await presentRevision(model)),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async handleUserEvent(
|
|
subscription: WebhookSubscription,
|
|
event: UserEvent
|
|
): Promise<void> {
|
|
const model = await User.findByPk(event.userId, {
|
|
paranoid: false,
|
|
});
|
|
|
|
await this.sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload: {
|
|
id: event.userId,
|
|
model: model && presentUser(model),
|
|
},
|
|
});
|
|
}
|
|
|
|
private async sendWebhook({
|
|
event,
|
|
subscription,
|
|
payload,
|
|
}: {
|
|
event: Event;
|
|
subscription: WebhookSubscription;
|
|
payload: WebhookPayload;
|
|
}) {
|
|
const delivery = await WebhookDelivery.create({
|
|
webhookSubscriptionId: subscription.id,
|
|
status: "pending",
|
|
});
|
|
|
|
let response, requestBody, requestHeaders, status;
|
|
try {
|
|
requestBody = presentWebhook({
|
|
event,
|
|
delivery,
|
|
payload,
|
|
});
|
|
requestHeaders = {
|
|
"Content-Type": "application/json",
|
|
"user-agent": `Outline-Webhooks${env.VERSION ? `/${env.VERSION}` : ""}`,
|
|
};
|
|
response = await fetch(subscription.url, {
|
|
method: "POST",
|
|
headers: requestHeaders,
|
|
body: JSON.stringify(requestBody),
|
|
});
|
|
status = response.ok ? "success" : "failed";
|
|
} catch (err) {
|
|
status = "failed";
|
|
}
|
|
|
|
await delivery.update({
|
|
status,
|
|
statusCode: response ? response.status : null,
|
|
requestBody,
|
|
requestHeaders,
|
|
responseBody: response ? await response.text() : "",
|
|
responseHeaders: response
|
|
? Object.fromEntries(response.headers.entries())
|
|
: {},
|
|
});
|
|
|
|
if (response && !response.ok) {
|
|
const recentDeliveries = await WebhookDelivery.findAll({
|
|
where: {
|
|
webhookSubscriptionId: subscription.id,
|
|
},
|
|
order: [["createdAt", "DESC"]],
|
|
limit: 25,
|
|
});
|
|
|
|
const allFailed = recentDeliveries.every(
|
|
(delivery) => delivery.status === "failed"
|
|
);
|
|
|
|
if (recentDeliveries.length === 25 && allFailed) {
|
|
await subscription.update({ enabled: false });
|
|
}
|
|
}
|
|
}
|
|
}
|