From 10f86ed2189c62345511575169428dc3ec81a00d Mon Sep 17 00:00:00 2001 From: Tom Moor Date: Wed, 29 Jun 2022 08:44:50 +0300 Subject: [PATCH] feat: Webhooks (#3691) * Webhooks (#3607) * Get the migration and the model setup. Also make the sample env file a bit easier to use. Now just requires setting a SECRET_KEY and besides that will boot up from the sample * WIP: Start getting a Webhook page created. Just the skeleton state right now * WIP: Getting a form created to create webhooks, need to bring in react-hook-forms now * WIP: Get library installed and make TS happy * Get a few checkboxes ready to go * Get creating and destroying working with a decent start to a frontend * Didn't mean to enable this * Remove eslint and fix other random typescript issue * Rename some events to be more realistic * Revert these changes * PR review comments around policies. Also make sure this inherits from IdModel so it actually gets an id * Allow any admin on the team to edit webhooks * Start sending some webhooks for some User events * Make sure the URL is valid * Start recording webhook deliveries * Make sure to verify if the subscription is for the type of event we are looking at * Refactor sending Webhooks and follow better webhook schema This creates a presenter to unify the format of webhooks. We also extract the sending of webhooks and recording their deliveries to a method than can be used by each of the different event type methods We also add a status to WebhookDelivery since we need to save the record before we make the HTTP request to get its id. Then once we make the request and get a response we can update the delivery with the HTTP info * Turn off a subscription that has failed for the last 25 deliveries * Get a first spec passing. Found a bug in my returning of promises so good to patch that up now * This looks nicer * Get some tests added for the processor * Add cron task to delete older webhooks * Add Document Events to the Processor * Revisions, FileOperations and Collections * Get all the server side events added to the processor and make Typescript make sure they are all accounted for * Get all the events added to the Frontend and work on styling them a bit, still needs some love though * Get UI styled up a bit * Get events wired up for webhook subscriptions * Get delete events working and test at least one variant of them * Get deletes working and actually make sure to send the model id in the webhook * Remove webhook secrets from this slice * Add disabled label for subscriptions that are disabled * Make sure to cascade the delete * Reorg this file a bit * Fix association * I removed secret for the moment * Apply Copy changes from PR Review Co-authored-by: Tom Moor * Actually apply the copy changes TIL that if you Resolve a conversation it _also_ removes the 'staged suggestion' from your list on Github Co-authored-by: Tom Moor * Update app/scenes/Settings/Webhooks.tsx Missed this copy change before Co-authored-by: Tom Moor * Add disabled as yellow badge * Resolve frontend comments * Fixup Schema a bit and remove the dependency on the subscription * Add test to make sure we don't disable until there are enough failures, and fix code to actually do that. Also some test fixes from the json response shape changes * Fix WebhookDeliveries to store the responses as Text instead of blobs * Switch to text better for response bodies, this is using the helpers better and makes the code read better * Move the logic to a task but run in through the processor cause the tests expect that right now, moving the tests over next * Split up the tests and actually enqueue the events from the WebhookProcessor instead of doing them inline * Allow any team admin to see any webhook subscription for the team * Add the indexes based on our lookup patterns * Run eslint --fix to fix auto correct issues from when I tried to use Github to merge copy changes * Allow subscriptions to be edited after creation * Types caught that I didn't add the new event to the webhook processor, also added it to the frontend here * I think this will get these into the translations file * Catch a few more translations, use styled components better and remove usage of webhook subscription in the copy Co-authored-by: Tom Moor * fix: tsc fix: Document model payload empty * fix: Revision webhook payload Add custom UA for hooks * Add webhooks icon, move under Integrations settings Some spacing fixes * Add actorId to webhook payloads * Add View and ApiKey event types * Spacing tweaks, fix team payload * fix: Webhook not disabled after 25 failures * fix: Enable webhook when editing if previously disabled * fix: Correctly store response headers * fix: Error in json/parsing/presentation results in hanging 'pending' webhook delivery * fix: Awkward payload for users.invite webhook * Add BaseEvent, ShareEvent * fix: Add share events to form * fix: Move webhook delivery cleanup to single DB call Remove some unused abstraction * Add user, collection, group context to membership webhook events Some associated refactoring Co-authored-by: Corey Alexander --- app/components/Input.tsx | 59 +- app/components/withStores.tsx | 8 +- app/hooks/useAuthorizedSettingsConfig.ts | 22 +- app/models/WebhookSubscription.ts | 27 + app/scenes/Settings/Webhooks.tsx | 69 +++ .../WebhookSubscriptionDeleteDialog.tsx | 34 ++ .../components/WebhookSubscriptionEdit.tsx | 57 ++ .../components/WebhookSubscriptionForm.tsx | 284 +++++++++ .../WebhookSubscriptionListItem.tsx | 89 +++ .../components/WebhookSubscriptionNew.tsx | 51 ++ app/stores/RootStore.ts | 4 + app/stores/WebhookSubscriptionStore.ts | 18 + app/test/setup.ts | 2 +- package.json | 3 +- server/commands/userInviter.ts | 1 + server/env.ts | 10 + server/logging/tracing.ts | 4 +- ...20521164111-create-webhook-subscription.js | 68 +++ ...220606031139-create-webhook-delivieries.js | 73 +++ server/models/CollectionGroup.ts | 17 + server/models/CollectionUser.ts | 17 + server/models/Event.ts | 2 + server/models/GroupUser.ts | 17 + server/models/Team.ts | 3 + server/models/View.ts | 12 + server/models/WebhookDelivery.ts | 53 ++ server/models/WebhookSubscription.ts | 70 +++ server/models/index.ts | 4 + server/policies/index.ts | 1 + server/policies/webhookSubscription.ts | 35 ++ server/presenters/groupMembership.ts | 9 +- server/presenters/index.ts | 4 + server/presenters/user.ts | 5 +- server/presenters/webhook.ts | 38 ++ server/presenters/webhookSubscription.ts | 13 + server/queues/processors/BaseProcessor.ts | 2 +- .../processors/NotificationsProcessor.test.ts | 9 + .../processors/WebhookProcessor.test.ts | 96 +++ server/queues/processors/WebhookProcessor.ts | 27 + .../CleanupWebhookDeliveriesTask.test.ts | 33 ++ .../tasks/CleanupWebhookDeliveriesTask.ts | 28 + .../queues/tasks/DeliverWebhookTask.test.ts | 202 +++++++ server/queues/tasks/DeliverWebhookTask.ts | 560 ++++++++++++++++++ server/routes/api/auth.ts | 6 +- server/routes/api/cron.ts | 3 + server/routes/api/groups.ts | 12 +- server/routes/api/index.ts | 2 + server/routes/api/views.ts | 1 + server/routes/api/webhookSubscriptions.ts | 137 +++++ server/test/factories.ts | 57 ++ server/types.ts | 389 ++++++------ shared/i18n/locales/en_US/translation.json | 18 + yarn.lock | 13 +- 53 files changed, 2531 insertions(+), 247 deletions(-) create mode 100644 app/models/WebhookSubscription.ts create mode 100644 app/scenes/Settings/Webhooks.tsx create mode 100644 app/scenes/Settings/components/WebhookSubscriptionDeleteDialog.tsx create mode 100644 app/scenes/Settings/components/WebhookSubscriptionEdit.tsx create mode 100644 app/scenes/Settings/components/WebhookSubscriptionForm.tsx create mode 100644 app/scenes/Settings/components/WebhookSubscriptionListItem.tsx create mode 100644 app/scenes/Settings/components/WebhookSubscriptionNew.tsx create mode 100644 app/stores/WebhookSubscriptionStore.ts create mode 100644 server/migrations/20220521164111-create-webhook-subscription.js create mode 100644 server/migrations/20220606031139-create-webhook-delivieries.js create mode 100644 server/models/WebhookDelivery.ts create mode 100644 server/models/WebhookSubscription.ts create mode 100644 server/policies/webhookSubscription.ts create mode 100644 server/presenters/webhook.ts create mode 100644 server/presenters/webhookSubscription.ts create mode 100644 server/queues/processors/WebhookProcessor.test.ts create mode 100644 server/queues/processors/WebhookProcessor.ts create mode 100644 server/queues/tasks/CleanupWebhookDeliveriesTask.test.ts create mode 100644 server/queues/tasks/CleanupWebhookDeliveriesTask.ts create mode 100644 server/queues/tasks/DeliverWebhookTask.test.ts create mode 100644 server/queues/tasks/DeliverWebhookTask.ts create mode 100644 server/routes/api/webhookSubscriptions.ts diff --git a/app/components/Input.tsx b/app/components/Input.tsx index 014b097e4..41d15a069 100644 --- a/app/components/Input.tsx +++ b/app/components/Input.tsx @@ -104,31 +104,17 @@ export const LabelText = styled.div` display: inline-block; `; -export type Props = Omit, "onChange"> & { +export type Props = React.InputHTMLAttributes< + HTMLInputElement | HTMLTextAreaElement +> & { type?: "text" | "email" | "checkbox" | "search" | "textarea"; - value?: string; - label?: string; - className?: string; labelHidden?: boolean; + label?: string; flex?: boolean; short?: boolean; margin?: string | number; icon?: React.ReactNode; - name?: string; - pattern?: string; - minLength?: number; - maxLength?: number; - autoFocus?: boolean; - autoComplete?: boolean | string; - readOnly?: boolean; - required?: boolean; - disabled?: boolean; - placeholder?: string; - onChange?: ( - ev: React.ChangeEvent - ) => unknown; - innerRef?: React.RefObject; - onKeyDown?: (ev: React.KeyboardEvent) => unknown; + innerRef?: React.Ref; onFocus?: (ev: React.SyntheticEvent) => unknown; onBlur?: (ev: React.SyntheticEvent) => unknown; }; @@ -171,8 +157,6 @@ class Input extends React.Component { ...rest } = this.props; - const InputComponent: React.ComponentType = - type === "textarea" ? RealTextarea : RealInput; const wrappedLabel = {label}; return ( @@ -186,15 +170,24 @@ class Input extends React.Component { ))} {icon && {icon}} - + {type === "textarea" ? ( + + ) : ( + + )} @@ -202,4 +195,10 @@ class Input extends React.Component { } } +export const ReactHookWrappedInput = React.forwardRef( + (props: Omit, ref: React.Ref) => { + return ; + } +); + export default Input; diff --git a/app/components/withStores.tsx b/app/components/withStores.tsx index 90792363f..1a685fa39 100644 --- a/app/components/withStores.tsx +++ b/app/components/withStores.tsx @@ -6,15 +6,13 @@ import useStores from "~/hooks/useStores"; type StoreProps = keyof RootStore; function withStores< - P extends React.ComponentType & RootStore>, + P extends React.ComponentType, ResolvedProps = JSX.LibraryManagedAttributes< P, Omit, StoreProps> > ->(WrappedComponent: P): React.FC> { - const ComponentWithStore = ( - props: Omit, StoreProps> - ) => { +>(WrappedComponent: P): React.FC { + const ComponentWithStore = (props: ResolvedProps) => { const stores = useStores(); return ; }; diff --git a/app/hooks/useAuthorizedSettingsConfig.ts b/app/hooks/useAuthorizedSettingsConfig.ts index f70def539..b542ee696 100644 --- a/app/hooks/useAuthorizedSettingsConfig.ts +++ b/app/hooks/useAuthorizedSettingsConfig.ts @@ -10,6 +10,7 @@ import { TeamIcon, BeakerIcon, DownloadIcon, + WebhooksIcon, } from "outline-icons"; import React from "react"; import { useTranslation } from "react-i18next"; @@ -25,6 +26,7 @@ import Security from "~/scenes/Settings/Security"; import Shares from "~/scenes/Settings/Shares"; import Slack from "~/scenes/Settings/Slack"; import Tokens from "~/scenes/Settings/Tokens"; +import Webhooks from "~/scenes/Settings/Webhooks"; import Zapier from "~/scenes/Settings/Zapier"; import SlackIcon from "~/components/SlackIcon"; import ZapierIcon from "~/components/ZapierIcon"; @@ -46,6 +48,7 @@ type SettingsPage = | "Shares" | "Import" | "Export" + | "Webhooks" | "Slack" | "Zapier"; @@ -158,7 +161,15 @@ const useAuthorizedSettingsConfig = () => { group: t("Team"), icon: DownloadIcon, }, - // Intergrations + // Integrations + Webhooks: { + name: t("Webhooks"), + path: "/settings/webhooks", + component: Webhooks, + enabled: can.createWebhookSubscription, + group: t("Integrations"), + icon: WebhooksIcon, + }, Slack: { name: "Slack", path: "/settings/integrations/slack", @@ -176,7 +187,14 @@ const useAuthorizedSettingsConfig = () => { icon: ZapierIcon, }, }), - [can.createApiKey, can.export, can.manage, can.update, t] + [ + can.createApiKey, + can.createWebhookSubscription, + can.export, + can.manage, + can.update, + t, + ] ); const enabledConfigs = React.useMemo( diff --git a/app/models/WebhookSubscription.ts b/app/models/WebhookSubscription.ts new file mode 100644 index 000000000..be7c73320 --- /dev/null +++ b/app/models/WebhookSubscription.ts @@ -0,0 +1,27 @@ +import { observable } from "mobx"; +import BaseModel from "./BaseModel"; +import Field from "./decorators/Field"; + +class WebhookSubscription extends BaseModel { + @Field + @observable + id: string; + + @Field + @observable + name: string; + + @Field + @observable + url: string; + + @Field + @observable + enabled: boolean; + + @Field + @observable + events: string[]; +} + +export default WebhookSubscription; diff --git a/app/scenes/Settings/Webhooks.tsx b/app/scenes/Settings/Webhooks.tsx new file mode 100644 index 000000000..322d003f1 --- /dev/null +++ b/app/scenes/Settings/Webhooks.tsx @@ -0,0 +1,69 @@ +import { observer } from "mobx-react"; +import { WebhooksIcon } from "outline-icons"; +import * as React from "react"; +import { useTranslation, Trans } from "react-i18next"; +import WebhookSubscription from "~/models/WebhookSubscription"; +import { Action } from "~/components/Actions"; +import Button from "~/components/Button"; +import Heading from "~/components/Heading"; +import Modal from "~/components/Modal"; +import PaginatedList from "~/components/PaginatedList"; +import Scene from "~/components/Scene"; +import Subheading from "~/components/Subheading"; +import Text from "~/components/Text"; +import useBoolean from "~/hooks/useBoolean"; +import useCurrentTeam from "~/hooks/useCurrentTeam"; +import usePolicy from "~/hooks/usePolicy"; +import useStores from "~/hooks/useStores"; +import WebhookSubscriptionListItem from "./components/WebhookSubscriptionListItem"; +import WebhookSubscriptionNew from "./components/WebhookSubscriptionNew"; + +function Webhooks() { + const team = useCurrentTeam(); + const { t } = useTranslation(); + const { webhookSubscriptions } = useStores(); + const [newModalOpen, handleNewModalOpen, handleNewModalClose] = useBoolean(); + const can = usePolicy(team.id); + + return ( + } + actions={ + <> + {can.createWebhookSubscription && ( + + + + ); +} + +export default WebhookSubscriptionForm; diff --git a/app/scenes/Settings/components/WebhookSubscriptionListItem.tsx b/app/scenes/Settings/components/WebhookSubscriptionListItem.tsx new file mode 100644 index 000000000..7a05dac32 --- /dev/null +++ b/app/scenes/Settings/components/WebhookSubscriptionListItem.tsx @@ -0,0 +1,89 @@ +import { EditIcon, TrashIcon } from "outline-icons"; +import * as React from "react"; +import { useTranslation } from "react-i18next"; +import styled from "styled-components"; +import WebhookSubscription from "~/models/WebhookSubscription"; +import Badge from "~/components/Badge"; +import Button from "~/components/Button"; +import ListItem from "~/components/List/Item"; +import Modal from "~/components/Modal"; +import useBoolean from "~/hooks/useBoolean"; +import useStores from "~/hooks/useStores"; +import WebhookSubscriptionRevokeDialog from "./WebhookSubscriptionDeleteDialog"; +import WebhookSubscriptionEdit from "./WebhookSubscriptionEdit"; + +type Props = { + webhook: WebhookSubscription; +}; + +const WebhookSubscriptionListItem = ({ webhook }: Props) => { + const { t } = useTranslation(); + const { dialogs } = useStores(); + const [ + editModalOpen, + handleEditModalOpen, + handleEditModalClose, + ] = useBoolean(); + + const showDeletionConfirmation = React.useCallback(() => { + dialogs.openModal({ + title: t("Delete webhook"), + isCentered: true, + content: ( + + ), + }); + }, [t, dialogs, webhook]); + + return ( + + {webhook.name} + {!webhook.enabled && ( + {t("Disabled")} + )} + + } + subtitle={ + <> + {t("Subscribed events")}: {webhook.events.join(", ")} + + } + actions={ + <> + + + + + + + } + /> + ); +}; + +const StyledBadge = styled(Badge)` + position: absolute; +`; + +export default WebhookSubscriptionListItem; diff --git a/app/scenes/Settings/components/WebhookSubscriptionNew.tsx b/app/scenes/Settings/components/WebhookSubscriptionNew.tsx new file mode 100644 index 000000000..2e1167c88 --- /dev/null +++ b/app/scenes/Settings/components/WebhookSubscriptionNew.tsx @@ -0,0 +1,51 @@ +import * as React from "react"; +import { useTranslation } from "react-i18next"; +import useStores from "~/hooks/useStores"; +import useToasts from "~/hooks/useToasts"; +import WebhookSubscriptionForm from "./WebhookSubscriptionForm"; + +type Props = { + onSubmit: () => void; +}; + +interface FormData { + name: string; + url: string; + events: string[]; +} + +function WebhookSubscriptionNew({ onSubmit }: Props) { + const { webhookSubscriptions } = useStores(); + const { showToast } = useToasts(); + const { t } = useTranslation(); + + const handleSubmit = React.useCallback( + async (data: FormData) => { + try { + const events = Array.isArray(data.events) ? data.events : [data.events]; + + const toSend = { + ...data, + events, + }; + + await webhookSubscriptions.create(toSend); + showToast( + t("Webhook created", { + type: "success", + }) + ); + onSubmit(); + } catch (err) { + showToast(err.message, { + type: "error", + }); + } + }, + [t, showToast, onSubmit, webhookSubscriptions] + ); + + return ; +} + +export default WebhookSubscriptionNew; diff --git a/app/stores/RootStore.ts b/app/stores/RootStore.ts index 8693358eb..88f8ac1f4 100644 --- a/app/stores/RootStore.ts +++ b/app/stores/RootStore.ts @@ -22,6 +22,7 @@ import ToastsStore from "./ToastsStore"; import UiStore from "./UiStore"; import UsersStore from "./UsersStore"; import ViewsStore from "./ViewsStore"; +import WebhookSubscriptionsStore from "./WebhookSubscriptionStore"; export default class RootStore { apiKeys: ApiKeysStore; @@ -48,6 +49,7 @@ export default class RootStore { views: ViewsStore; toasts: ToastsStore; fileOperations: FileOperationsStore; + webhookSubscriptions: WebhookSubscriptionsStore; constructor() { // PoliciesStore must be initialized before AuthStore @@ -75,6 +77,7 @@ export default class RootStore { this.views = new ViewsStore(this); this.fileOperations = new FileOperationsStore(this); this.toasts = new ToastsStore(); + this.webhookSubscriptions = new WebhookSubscriptionsStore(this); } logout() { @@ -100,5 +103,6 @@ export default class RootStore { // this.ui omitted to keep ui settings between sessions this.users.clear(); this.views.clear(); + this.webhookSubscriptions.clear(); } } diff --git a/app/stores/WebhookSubscriptionStore.ts b/app/stores/WebhookSubscriptionStore.ts new file mode 100644 index 000000000..e2c5cce49 --- /dev/null +++ b/app/stores/WebhookSubscriptionStore.ts @@ -0,0 +1,18 @@ +import WebhookSubscription from "~/models/WebhookSubscription"; +import BaseStore, { RPCAction } from "./BaseStore"; +import RootStore from "./RootStore"; + +export default class WebhookSubscriptionsStore extends BaseStore< + WebhookSubscription +> { + actions = [ + RPCAction.List, + RPCAction.Create, + RPCAction.Delete, + RPCAction.Update, + ]; + + constructor(rootStore: RootStore) { + super(rootStore, WebhookSubscription); + } +} diff --git a/app/test/setup.ts b/app/test/setup.ts index 6ca766d43..64b8c6ee6 100644 --- a/app/test/setup.ts +++ b/app/test/setup.ts @@ -13,4 +13,4 @@ Enzyme.configure({ global.localStorage = localStorage; -require("jest-fetch-mock").enableMocks(); \ No newline at end of file +require("jest-fetch-mock").enableMocks(); diff --git a/package.json b/package.json index dc556fcc0..3fb97c083 100644 --- a/package.json +++ b/package.json @@ -133,7 +133,7 @@ "mobx-react": "^6.3.1", "natural-sort": "^1.0.0", "nodemailer": "^6.6.1", - "outline-icons": "^1.42.0", + "outline-icons": "^1.43.1", "oy-vey": "^0.10.0", "passport": "^0.4.1", "passport-google-oauth2": "^0.2.0", @@ -168,6 +168,7 @@ "react-dom": "^17.0.2", "react-dropzone": "^11.3.2", "react-helmet": "^6.1.0", + "react-hook-form": "^7.31.2", "react-i18next": "^11.16.6", "react-medium-image-zoom": "^3.1.3", "react-merge-refs": "^1.1.0", diff --git a/server/commands/userInviter.ts b/server/commands/userInviter.ts index e4c740ece..6eabe0f76 100644 --- a/server/commands/userInviter.ts +++ b/server/commands/userInviter.ts @@ -73,6 +73,7 @@ export default async function userInviter({ name: "users.invite", actorId: user.id, teamId: user.teamId, + userId: newUser.id, data: { email: invite.email, name: invite.name, diff --git a/server/env.ts b/server/env.ts index 0c9cc7d65..3368db972 100644 --- a/server/env.ts +++ b/server/env.ts @@ -485,6 +485,16 @@ export class Environment { */ public OIDC_SCOPES = process.env.OIDC_SCOPES ?? "openid profile email"; + /** + * A string representing the version of the software. + * + * SOURCE_COMMIT is used by Docker Hub + * SOURCE_VERSION is used by Heroku + */ + public VERSION = this.toOptionalString( + process.env.SOURCE_COMMIT || process.env.SOURCE_VERSION + ); + private toOptionalString(value: string | undefined) { return value ? value : undefined; } diff --git a/server/logging/tracing.ts b/server/logging/tracing.ts index b3ee8b507..ac06a9e0a 100644 --- a/server/logging/tracing.ts +++ b/server/logging/tracing.ts @@ -8,9 +8,7 @@ export * as APM from "@theo.gravity/datadog-apm"; if (env.DD_API_KEY) { init( { - // SOURCE_COMMIT is used by Docker Hub - // SOURCE_VERSION is used by Heroku - version: process.env.SOURCE_COMMIT || process.env.SOURCE_VERSION, + version: env.VERSION, service: process.env.DD_SERVICE || "outline", }, { diff --git a/server/migrations/20220521164111-create-webhook-subscription.js b/server/migrations/20220521164111-create-webhook-subscription.js new file mode 100644 index 000000000..6908df16e --- /dev/null +++ b/server/migrations/20220521164111-create-webhook-subscription.js @@ -0,0 +1,68 @@ +"use strict"; +module.exports = { + up: async (queryInterface, Sequelize) => { + await queryInterface.sequelize.transaction(async (transaction) => { + await queryInterface.createTable( + "webhook_subscriptions", + { + id: { + type: Sequelize.UUID, + allowNull: false, + primaryKey: true, + }, + teamId: { + type: Sequelize.UUID, + allowNull: false, + references: { + model: "teams", + }, + }, + createdById: { + type: Sequelize.UUID, + allowNull: false, + references: { + model: "users", + }, + }, + url: { + type: Sequelize.STRING, + allowNull: false, + }, + enabled: { + type: Sequelize.BOOLEAN, + allowNull: false, + }, + name: { + type: Sequelize.STRING, + allowNull: false, + }, + events: { + type: Sequelize.ARRAY(Sequelize.STRING), + allowNull: false, + }, + createdAt: { + allowNull: false, + type: Sequelize.DATE, + }, + updatedAt: { + allowNull: false, + type: Sequelize.DATE, + }, + }, + { transaction } + ); + + await queryInterface.addIndex( + "webhook_subscriptions", + ["teamId", "enabled"], + { + name: "webhook_subscriptions_team_id_enabled", + transaction, + } + ); + }); + }, + down: async (queryInterface, Sequelize) => { + return queryInterface.dropTable("webhook_subscriptions"); + }, +}; diff --git a/server/migrations/20220606031139-create-webhook-delivieries.js b/server/migrations/20220606031139-create-webhook-delivieries.js new file mode 100644 index 000000000..9d34d9113 --- /dev/null +++ b/server/migrations/20220606031139-create-webhook-delivieries.js @@ -0,0 +1,73 @@ +"use strict"; +module.exports = { + up: async (queryInterface, Sequelize) => { + await queryInterface.sequelize.transaction(async (transaction) => { + await queryInterface.createTable( + "webhook_deliveries", + { + id: { + type: Sequelize.UUID, + allowNull: false, + primaryKey: true, + }, + webhookSubscriptionId: { + type: Sequelize.UUID, + allowNull: false, + onDelete: "cascade", + references: { + model: "webhook_subscriptions", + }, + }, + status: { + type: Sequelize.STRING, + allowNull: false, + }, + statusCode: { + type: Sequelize.INTEGER, + allowNull: true, + }, + requestBody: { + type: Sequelize.JSONB, + allowNull: true, + }, + requestHeaders: { + type: Sequelize.JSONB, + allowNull: true, + }, + responseBody: { + type: Sequelize.TEXT, + allowNull: true, + }, + responseHeaders: { + type: Sequelize.JSONB, + allowNull: true, + }, + createdAt: { + type: Sequelize.DATE, + allowNull: false, + }, + updatedAt: { + type: Sequelize.DATE, + allowNull: false, + }, + }, + { transaction } + ); + await queryInterface.addIndex( + "webhook_deliveries", + ["webhookSubscriptionId"], + { + name: "webhook_deliveries_webhook_subscription_id", + transaction, + } + ); + await queryInterface.addIndex("webhook_deliveries", ["createdAt"], { + name: "webhook_deliveries_createdAt", + transaction, + }); + }); + }, + down: async (queryInterface, Sequelize) => { + return queryInterface.dropTable("webhook_deliveries"); + }, +}; diff --git a/server/models/CollectionGroup.ts b/server/models/CollectionGroup.ts index b6edc1dba..bb51a7ef2 100644 --- a/server/models/CollectionGroup.ts +++ b/server/models/CollectionGroup.ts @@ -6,6 +6,7 @@ import { IsIn, Table, DataType, + Scopes, } from "sequelize-typescript"; import Collection from "./Collection"; import Group from "./Group"; @@ -13,6 +14,22 @@ import User from "./User"; import BaseModel from "./base/BaseModel"; import Fix from "./decorators/Fix"; +@Scopes(() => ({ + withGroup: { + include: [ + { + association: "group", + }, + ], + }, + withCollection: { + include: [ + { + association: "collection", + }, + ], + }, +})) @Table({ tableName: "collection_groups", modelName: "collection_group" }) @Fix class CollectionGroup extends BaseModel { diff --git a/server/models/CollectionUser.ts b/server/models/CollectionUser.ts index b3a7b0dd8..7499e4a85 100644 --- a/server/models/CollectionUser.ts +++ b/server/models/CollectionUser.ts @@ -6,12 +6,29 @@ import { IsIn, Table, DataType, + Scopes, } from "sequelize-typescript"; import Collection from "./Collection"; import User from "./User"; import BaseModel from "./base/BaseModel"; import Fix from "./decorators/Fix"; +@Scopes(() => ({ + withUser: { + include: [ + { + association: "user", + }, + ], + }, + withCollection: { + include: [ + { + association: "collection", + }, + ], + }, +})) @Table({ tableName: "collection_users", modelName: "collection_user" }) @Fix class CollectionUser extends BaseModel { diff --git a/server/models/Event.ts b/server/models/Event.ts index 239696df2..4cfd5fe81 100644 --- a/server/models/Event.ts +++ b/server/models/Event.ts @@ -167,6 +167,8 @@ class Event extends IdModel { "users.suspend", "users.activate", "users.delete", + "webhook_subscriptions.create", + "webhook_subscriptions.delete", ]; } diff --git a/server/models/GroupUser.ts b/server/models/GroupUser.ts index 34f1898f3..7a343180b 100644 --- a/server/models/GroupUser.ts +++ b/server/models/GroupUser.ts @@ -5,6 +5,7 @@ import { Column, Table, DataType, + Scopes, } from "sequelize-typescript"; import Group from "./Group"; import User from "./User"; @@ -18,6 +19,22 @@ import Fix from "./decorators/Fix"; }, ], })) +@Scopes(() => ({ + withGroup: { + include: [ + { + association: "group", + }, + ], + }, + withUser: { + include: [ + { + association: "user", + }, + ], + }, +})) @Table({ tableName: "group_users", modelName: "group_user", paranoid: true }) @Fix class GroupUser extends BaseModel { diff --git a/server/models/Team.ts b/server/models/Team.ts index 5e86088fb..c79e9eb76 100644 --- a/server/models/Team.ts +++ b/server/models/Team.ts @@ -35,6 +35,9 @@ import Fix from "./decorators/Fix"; const readFile = util.promisify(fs.readFile); @Scopes(() => ({ + withDomains: { + include: [{ model: TeamDomain }], + }, withAuthenticationProviders: { include: [ { diff --git a/server/models/View.ts b/server/models/View.ts index c9acb37ad..de6df60d0 100644 --- a/server/models/View.ts +++ b/server/models/View.ts @@ -7,6 +7,7 @@ import { ForeignKey, Table, DataType, + Scopes, } from "sequelize-typescript"; import { USER_PRESENCE_INTERVAL } from "@shared/constants"; import Document from "./Document"; @@ -14,6 +15,17 @@ import User from "./User"; import IdModel from "./base/IdModel"; import Fix from "./decorators/Fix"; +@Scopes(() => ({ + withUser: () => ({ + include: [ + { + model: User, + required: true, + as: "user", + }, + ], + }), +})) @Table({ tableName: "views", modelName: "view" }) @Fix class View extends IdModel { diff --git a/server/models/WebhookDelivery.ts b/server/models/WebhookDelivery.ts new file mode 100644 index 000000000..7e3ffe9b9 --- /dev/null +++ b/server/models/WebhookDelivery.ts @@ -0,0 +1,53 @@ +import { + Column, + Table, + BelongsTo, + ForeignKey, + NotEmpty, + DataType, + IsIn, +} from "sequelize-typescript"; +import WebhookSubscription from "./WebhookSubscription"; +import IdModel from "./base/IdModel"; +import Fix from "./decorators/Fix"; + +@Table({ + tableName: "webhook_deliveries", + modelName: "webhook_delivery", +}) +@Fix +class WebhookDelivery extends IdModel { + @NotEmpty + @IsIn([["pending", "success", "failed"]]) + @Column(DataType.STRING) + status: "pending" | "success" | "failed"; + + @Column(DataType.INTEGER) + statusCode: number; + + @Column(DataType.JSONB) + requestBody: unknown; + + @Column(DataType.JSONB) + requestHeaders: Record; + + @Column(DataType.TEXT) + responseBody: string; + + @Column(DataType.JSONB) + responseHeaders: Record; + + @Column(DataType.DATE) + createdAt: Date; + + // associations + + @BelongsTo(() => WebhookSubscription, "webhookSubscriptionId") + webhookSubscription: WebhookSubscription; + + @ForeignKey(() => WebhookSubscription) + @Column + webhookSubscriptionId: string; +} + +export default WebhookDelivery; diff --git a/server/models/WebhookSubscription.ts b/server/models/WebhookSubscription.ts new file mode 100644 index 000000000..61ff420c8 --- /dev/null +++ b/server/models/WebhookSubscription.ts @@ -0,0 +1,70 @@ +import { bool } from "aws-sdk/clients/signer"; +import { + Column, + Table, + BelongsTo, + ForeignKey, + NotEmpty, + DataType, + IsUrl, +} from "sequelize-typescript"; +import { Event } from "@server/types"; +import Team from "./Team"; +import User from "./User"; +import IdModel from "./base/IdModel"; +import Fix from "./decorators/Fix"; + +@Table({ + tableName: "webhook_subscriptions", + modelName: "webhook_subscription", +}) +@Fix +class WebhookSubscription extends IdModel { + @NotEmpty + @Column + name: string; + + @IsUrl + @NotEmpty + @Column + url: string; + + @Column + enabled: boolean; + + @Column(DataType.ARRAY(DataType.STRING)) + events: string[]; + + // associations + + @BelongsTo(() => User, "createdById") + createdBy: User; + + @ForeignKey(() => User) + @Column + createdById: string; + + @BelongsTo(() => Team, "teamId") + team: Team; + + @ForeignKey(() => Team) + @Column + teamId: string; + + // methods + validForEvent = (event: Event): bool => { + if (this.events.length === 1 && this.events[0] === "*") { + return true; + } + + for (const e of this.events) { + if (e === event.name || event.name.startsWith(e + ".")) { + return true; + } + } + + return false; + }; +} + +export default WebhookSubscription; diff --git a/server/models/index.ts b/server/models/index.ts index 16558caf7..3ce2dc91b 100644 --- a/server/models/index.ts +++ b/server/models/index.ts @@ -49,3 +49,7 @@ export { default as User } from "./User"; export { default as UserAuthentication } from "./UserAuthentication"; export { default as View } from "./View"; + +export { default as WebhookSubscription } from "./WebhookSubscription"; + +export { default as WebhookDelivery } from "./WebhookDelivery"; diff --git a/server/policies/index.ts b/server/policies/index.ts index c2240889f..22147b4b7 100644 --- a/server/policies/index.ts +++ b/server/policies/index.ts @@ -21,6 +21,7 @@ import "./star"; import "./user"; import "./team"; import "./group"; +import "./webhookSubscription"; type Policy = Record; diff --git a/server/policies/webhookSubscription.ts b/server/policies/webhookSubscription.ts new file mode 100644 index 000000000..27a155333 --- /dev/null +++ b/server/policies/webhookSubscription.ts @@ -0,0 +1,35 @@ +import { User, Team, WebhookSubscription } from "@server/models"; +import { allow } from "./cancan"; + +allow(User, "listWebhookSubscription", Team, (user, team) => { + if (!team || user.isViewer || user.teamId !== team.id) { + return false; + } + + return user.isAdmin; +}); + +allow(User, "createWebhookSubscription", Team, (user, team) => { + if (!team || user.isViewer || user.teamId !== team.id) { + return false; + } + + return user.isAdmin; +}); + +allow( + User, + ["read", "update", "delete"], + WebhookSubscription, + (user, webhook): boolean => { + if (!user || !webhook) { + return false; + } + + if (!user.isAdmin) { + return false; + } + + return user.teamId === webhook.teamId; + } +); diff --git a/server/presenters/groupMembership.ts b/server/presenters/groupMembership.ts index 4c59607d2..cffe84d2a 100644 --- a/server/presenters/groupMembership.ts +++ b/server/presenters/groupMembership.ts @@ -5,14 +5,17 @@ type GroupMembership = { id: string; userId: string; groupId: string; - user: ReturnType; + user?: ReturnType; }; -export default (membership: GroupUser): GroupMembership => { +export default ( + membership: GroupUser, + options?: { includeUser: boolean } +): GroupMembership => { return { id: `${membership.userId}-${membership.groupId}`, userId: membership.userId, groupId: membership.groupId, - user: presentUser(membership.user), + user: options?.includeUser ? presentUser(membership.user) : undefined, }; }; diff --git a/server/presenters/index.ts b/server/presenters/index.ts index 24cc1bd63..a37327a7c 100644 --- a/server/presenters/index.ts +++ b/server/presenters/index.ts @@ -20,6 +20,8 @@ import presentStar from "./star"; import presentTeam from "./team"; import presentUser from "./user"; import presentView from "./view"; +import presentWebhook from "./webhook"; +import presentWebhookSubscription from "./webhookSubscription"; export { presentApiKey, @@ -44,4 +46,6 @@ export { presentPolicies, presentGroupMembership, presentCollectionGroupMembership, + presentWebhook, + presentWebhookSubscription, }; diff --git a/server/presenters/user.ts b/server/presenters/user.ts index 2c9e39e43..fdf8dbae3 100644 --- a/server/presenters/user.ts +++ b/server/presenters/user.ts @@ -20,10 +20,7 @@ type UserPresentation = { language?: string; }; -export default ( - user: User, - options: Options = {} -): UserPresentation | null | undefined => { +export default (user: User, options: Options = {}): UserPresentation => { const userData: UserPresentation = { id: user.id, name: user.name, diff --git a/server/presenters/webhook.ts b/server/presenters/webhook.ts new file mode 100644 index 000000000..4017f8d5e --- /dev/null +++ b/server/presenters/webhook.ts @@ -0,0 +1,38 @@ +import { WebhookDelivery } from "@server/models"; +import { Event } from "@server/types"; + +export interface WebhookPayload { + model: Record | null; + id: string; + [key: string]: unknown; +} + +interface WebhookProps { + event: Event; + delivery: WebhookDelivery; + payload: WebhookPayload; +} + +export interface WebhookPresentation { + id: string; + actorId: string; + webhookSubscriptionId: string; + event: string; + payload: WebhookPayload; + createdAt: Date; +} + +export default function present({ + event, + delivery, + payload, +}: WebhookProps): WebhookPresentation { + return { + id: delivery.id, + actorId: event.actorId, + webhookSubscriptionId: delivery.webhookSubscriptionId, + createdAt: delivery.createdAt, + event: event.name, + payload: payload, + }; +} diff --git a/server/presenters/webhookSubscription.ts b/server/presenters/webhookSubscription.ts new file mode 100644 index 000000000..d1cf73995 --- /dev/null +++ b/server/presenters/webhookSubscription.ts @@ -0,0 +1,13 @@ +import { WebhookSubscription } from "@server/models"; + +export default function present(webhook: WebhookSubscription) { + return { + id: webhook.id, + name: webhook.name, + url: webhook.url, + events: webhook.events, + enabled: webhook.enabled, + createdAt: webhook.createdAt, + updatedAt: webhook.updatedAt, + }; +} diff --git a/server/queues/processors/BaseProcessor.ts b/server/queues/processors/BaseProcessor.ts index 72c379ce0..6cf0c13f9 100644 --- a/server/queues/processors/BaseProcessor.ts +++ b/server/queues/processors/BaseProcessor.ts @@ -1,7 +1,7 @@ import { Event } from "@server/types"; export default abstract class BaseProcessor { - static applicableEvents: (Event["name"] | "*")[] = []; + static applicableEvents: Event["name"][] | ["*"] = []; public abstract perform(event: Event): Promise; } diff --git a/server/queues/processors/NotificationsProcessor.test.ts b/server/queues/processors/NotificationsProcessor.test.ts index e70a4fed8..e17aefbc4 100644 --- a/server/queues/processors/NotificationsProcessor.test.ts +++ b/server/queues/processors/NotificationsProcessor.test.ts @@ -118,6 +118,9 @@ describe("revisions.create", () => { documentId: document.id, collectionId: document.collectionId, teamId: document.teamId, + actorId: collaborator.id, + modelId: document.id, + ip, }); expect(DocumentNotificationEmail.schedule).toHaveBeenCalled(); }); @@ -142,6 +145,9 @@ describe("revisions.create", () => { documentId: document.id, collectionId: document.collectionId, teamId: document.teamId, + actorId: collaborator.id, + modelId: document.id, + ip, }); expect(DocumentNotificationEmail.schedule).not.toHaveBeenCalled(); }); @@ -163,6 +169,9 @@ describe("revisions.create", () => { documentId: document.id, collectionId: document.collectionId, teamId: document.teamId, + actorId: user.id, + modelId: document.id, + ip, }); expect(DocumentNotificationEmail.schedule).not.toHaveBeenCalled(); }); diff --git a/server/queues/processors/WebhookProcessor.test.ts b/server/queues/processors/WebhookProcessor.test.ts new file mode 100644 index 000000000..826d5c712 --- /dev/null +++ b/server/queues/processors/WebhookProcessor.test.ts @@ -0,0 +1,96 @@ +import { buildUser, buildWebhookSubscription } from "@server/test/factories"; +import { flushdb } from "@server/test/support"; +import { UserEvent } from "@server/types"; +import DeliverWebhookTask from "../tasks/DeliverWebhookTask"; +import WebhookProcessor from "./WebhookProcessor"; + +jest.mock("@server/queues/tasks/DeliverWebhookTask"); + +beforeEach(() => flushdb()); +beforeEach(() => { + jest.resetAllMocks(); +}); + +const ip = "127.0.0.1"; + +describe("WebhookProcessor", () => { + test("it schedules a delivery for the event", async () => { + const subscription = await buildWebhookSubscription({ + url: "http://example.com", + events: ["*"], + }); + const signedInUser = await buildUser({ teamId: subscription.teamId }); + const processor = new WebhookProcessor(); + + const event: UserEvent = { + name: "users.signin", + userId: signedInUser.id, + teamId: subscription.teamId, + actorId: signedInUser.id, + ip, + }; + + await processor.perform(event); + + expect(DeliverWebhookTask.schedule).toHaveBeenCalled(); + expect(DeliverWebhookTask.schedule).toHaveBeenCalledWith({ + event, + subscriptionId: subscription.id, + }); + }); + + test("not schedule a delivery when not subscribed to event", async () => { + const subscription = await buildWebhookSubscription({ + url: "http://example.com", + events: ["users.create"], + }); + const signedInUser = await buildUser({ teamId: subscription.teamId }); + const processor = new WebhookProcessor(); + const event: UserEvent = { + name: "users.signin", + userId: signedInUser.id, + teamId: subscription.teamId, + actorId: signedInUser.id, + ip, + }; + + await processor.perform(event); + + expect(DeliverWebhookTask.schedule).toHaveBeenCalledTimes(0); + }); + + test("it schedules a delivery for the event for each subscription", async () => { + const subscription = await buildWebhookSubscription({ + url: "http://example.com", + events: ["*"], + }); + const subscriptionTwo = await buildWebhookSubscription({ + url: "http://example.com", + teamId: subscription.teamId, + events: ["*"], + }); + const signedInUser = await buildUser({ teamId: subscription.teamId }); + const processor = new WebhookProcessor(); + + const event: UserEvent = { + name: "users.signin", + userId: signedInUser.id, + teamId: subscription.teamId, + actorId: signedInUser.id, + ip, + }; + + await processor.perform(event); + + expect(DeliverWebhookTask.schedule).toHaveBeenCalled(); + expect(DeliverWebhookTask.schedule).toHaveBeenCalledTimes(2); + expect(DeliverWebhookTask.schedule).toHaveBeenCalledWith({ + event, + subscriptionId: subscription.id, + }); + expect(DeliverWebhookTask.schedule).toHaveBeenCalledWith({ + event, + subscriptionId: subscriptionTwo.id, + }); + }); +}); diff --git a/server/queues/processors/WebhookProcessor.ts b/server/queues/processors/WebhookProcessor.ts new file mode 100644 index 000000000..cb9393b80 --- /dev/null +++ b/server/queues/processors/WebhookProcessor.ts @@ -0,0 +1,27 @@ +import { WebhookSubscription } from "@server/models"; +import { Event } from "@server/types"; +import DeliverWebhookTask from "../tasks/DeliverWebhookTask"; +import BaseProcessor from "./BaseProcessor"; + +export default class WebhookProcessor extends BaseProcessor { + static applicableEvents: ["*"] = ["*"]; + + async perform(event: Event) { + const webhookSubscriptions = await WebhookSubscription.findAll({ + where: { + enabled: true, + teamId: event.teamId, + }, + }); + + const applicableSubscriptions = webhookSubscriptions.filter((webhook) => + webhook.validForEvent(event) + ); + + await Promise.all( + applicableSubscriptions.map((subscription) => + DeliverWebhookTask.schedule({ event, subscriptionId: subscription.id }) + ) + ); + } +} diff --git a/server/queues/tasks/CleanupWebhookDeliveriesTask.test.ts b/server/queues/tasks/CleanupWebhookDeliveriesTask.test.ts new file mode 100644 index 000000000..8b45d3c03 --- /dev/null +++ b/server/queues/tasks/CleanupWebhookDeliveriesTask.test.ts @@ -0,0 +1,33 @@ +import { subDays } from "date-fns"; +import { WebhookDelivery } from "@server/models"; +import { buildWebhookDelivery } from "@server/test/factories"; +import { flushdb } from "@server/test/support"; +import CleanupWebhookDeliveriesTask from "./CleanupWebhookDeliveriesTask"; + +beforeEach(() => flushdb()); + +const deliveryExists = async (delivery: WebhookDelivery) => { + const results = await WebhookDelivery.findOne({ where: { id: delivery.id } }); + return !!results; +}; + +describe("CleanupWebookDeliveriesTask", () => { + it("should delete Webhook Deliveries older than 1 week", async () => { + const brandNewWebhookDelivery = await buildWebhookDelivery({ + createdAt: new Date(), + }); + const newishWebhookDelivery = await buildWebhookDelivery({ + createdAt: subDays(new Date(), 5), + }); + const oldWebhookDelivery = await buildWebhookDelivery({ + createdAt: subDays(new Date(), 8), + }); + + const task = new CleanupWebhookDeliveriesTask(); + await task.perform(); + + expect(await deliveryExists(brandNewWebhookDelivery)).toBe(true); + expect(await deliveryExists(newishWebhookDelivery)).toBe(true); + expect(await deliveryExists(oldWebhookDelivery)).toBe(false); + }); +}); diff --git a/server/queues/tasks/CleanupWebhookDeliveriesTask.ts b/server/queues/tasks/CleanupWebhookDeliveriesTask.ts new file mode 100644 index 000000000..631a3c3e9 --- /dev/null +++ b/server/queues/tasks/CleanupWebhookDeliveriesTask.ts @@ -0,0 +1,28 @@ +import { subDays } from "date-fns"; +import { Op } from "sequelize"; +import Logger from "@server/logging/Logger"; +import { WebhookDelivery } from "@server/models"; +import BaseTask, { TaskPriority } from "./BaseTask"; + +type Props = void; + +export default class CleanupWebhookDeliveriesTask extends BaseTask { + public async perform(_: Props) { + Logger.info("task", `Deleting WebhookDeliveries older than one week…`); + const count = await WebhookDelivery.unscoped().destroy({ + where: { + createdAt: { + [Op.lt]: subDays(new Date(), 7), + }, + }, + }); + Logger.info("task", `${count} old WebhookDeliveries deleted.`); + } + + public get options() { + return { + attempts: 1, + priority: TaskPriority.Background, + }; + } +} diff --git a/server/queues/tasks/DeliverWebhookTask.test.ts b/server/queues/tasks/DeliverWebhookTask.test.ts new file mode 100644 index 000000000..348ee8c6e --- /dev/null +++ b/server/queues/tasks/DeliverWebhookTask.test.ts @@ -0,0 +1,202 @@ +import fetchMock from "jest-fetch-mock"; +import { v4 as uuidv4 } from "uuid"; +import { WebhookDelivery } from "@server/models"; +import { + buildUser, + buildWebhookDelivery, + buildWebhookSubscription, +} from "@server/test/factories"; +import { flushdb } from "@server/test/support"; +import { UserEvent } from "@server/types"; +import DeliverWebhookTask from "./DeliverWebhookTask"; + +beforeEach(() => flushdb()); +beforeEach(() => { + jest.resetAllMocks(); + fetchMock.resetMocks(); +}); + +const ip = "127.0.0.1"; + +fetchMock.enableMocks(); + +describe("DeliverWebhookTask", () => { + test("should hit the subscription url and record a delivery", async () => { + const subscription = await buildWebhookSubscription({ + url: "http://example.com", + events: ["*"], + }); + const signedInUser = await buildUser({ teamId: subscription.teamId }); + const processor = new DeliverWebhookTask(); + + fetchMock.mockResponse("SUCCESS", { status: 200 }); + + const event: UserEvent = { + name: "users.signin", + userId: signedInUser.id, + teamId: subscription.teamId, + actorId: signedInUser.id, + ip, + }; + await processor.perform({ + subscriptionId: subscription.id, + event, + }); + + expect(fetchMock).toHaveBeenCalledTimes(1); + expect(fetchMock).toHaveBeenCalledWith( + "http://example.com", + expect.anything() + ); + const parsedBody = JSON.parse( + fetchMock.mock.calls[0]![1]!.body!.toString() + ); + expect(parsedBody.webhookSubscriptionId).toBe(subscription.id); + expect(parsedBody.event).toBe("users.signin"); + expect(parsedBody.payload.id).toBe(signedInUser.id); + expect(parsedBody.payload.model).toBeDefined(); + + const deliveries = await WebhookDelivery.findAll({ + where: { webhookSubscriptionId: subscription.id }, + }); + expect(deliveries.length).toBe(1); + + const delivery = deliveries[0]; + expect(delivery.status).toBe("success"); + expect(delivery.statusCode).toBe(200); + expect(delivery.responseBody).toEqual("SUCCESS"); + }); + + test("should hit the subscription url when the eventing model doesn't exist", async () => { + const subscription = await buildWebhookSubscription({ + url: "http://example.com", + events: ["*"], + }); + const deletedUserId = uuidv4(); + const signedInUser = await buildUser({ teamId: subscription.teamId }); + + const task = new DeliverWebhookTask(); + const event: UserEvent = { + name: "users.delete", + userId: deletedUserId, + teamId: subscription.teamId, + actorId: signedInUser.id, + ip, + }; + + await task.perform({ + event, + subscriptionId: subscription.id, + }); + + expect(fetchMock).toHaveBeenCalledTimes(1); + expect(fetchMock).toHaveBeenCalledWith( + "http://example.com", + expect.anything() + ); + const parsedBody = JSON.parse( + fetchMock.mock.calls[0]![1]!.body!.toString() + ); + expect(parsedBody.webhookSubscriptionId).toBe(subscription.id); + expect(parsedBody.event).toBe("users.delete"); + expect(parsedBody.payload.id).toBe(deletedUserId); + + const deliveries = await WebhookDelivery.findAll({ + where: { webhookSubscriptionId: subscription.id }, + }); + expect(deliveries.length).toBe(1); + + const delivery = deliveries[0]; + expect(delivery.status).toBe("success"); + expect(delivery.statusCode).toBe(200); + expect(delivery.responseBody).toBeDefined(); + }); + + test("should mark delivery as failed if post fails", async () => { + const subscription = await buildWebhookSubscription({ + url: "http://example.com", + events: ["*"], + }); + + fetchMock.mockResponse("FAILED", { status: 500 }); + + const signedInUser = await buildUser({ teamId: subscription.teamId }); + const task = new DeliverWebhookTask(); + + const event: UserEvent = { + name: "users.signin", + userId: signedInUser.id, + teamId: subscription.teamId, + actorId: signedInUser.id, + ip, + }; + + await task.perform({ + event, + subscriptionId: subscription.id, + }); + + await subscription.reload(); + + expect(subscription.enabled).toBe(true); + + const deliveries = await WebhookDelivery.findAll({ + where: { webhookSubscriptionId: subscription.id }, + }); + expect(deliveries.length).toBe(1); + + const delivery = deliveries[0]; + expect(delivery.status).toBe("failed"); + expect(delivery.statusCode).toBe(500); + expect(delivery.responseBody).toBeDefined(); + expect(delivery.responseBody).toEqual("FAILED"); + }); + + test("should disable the subscription if past deliveries failed", async () => { + const subscription = await buildWebhookSubscription({ + url: "http://example.com", + events: ["*"], + }); + for (let i = 0; i < 25; i++) { + await buildWebhookDelivery({ + webhookSubscriptionId: subscription.id, + status: "failed", + }); + } + + fetchMock.mockResponse(JSON.stringify({ message: "Failure" }), { + status: 500, + }); + + const signedInUser = await buildUser({ teamId: subscription.teamId }); + const task = new DeliverWebhookTask(); + + const event: UserEvent = { + name: "users.signin", + userId: signedInUser.id, + teamId: subscription.teamId, + actorId: signedInUser.id, + ip, + }; + + await task.perform({ + event, + subscriptionId: subscription.id, + }); + + await subscription.reload(); + + expect(subscription.enabled).toBe(false); + + const deliveries = await WebhookDelivery.findAll({ + where: { webhookSubscriptionId: subscription.id }, + order: [["createdAt", "DESC"]], + }); + expect(deliveries.length).toBe(26); + + const delivery = deliveries[0]; + expect(delivery.status).toBe("failed"); + expect(delivery.statusCode).toBe(500); + expect(delivery.responseBody).toEqual('{"message":"Failure"}'); + }); +}); diff --git a/server/queues/tasks/DeliverWebhookTask.ts b/server/queues/tasks/DeliverWebhookTask.ts new file mode 100644 index 000000000..32fd6d7dc --- /dev/null +++ b/server/queues/tasks/DeliverWebhookTask.ts @@ -0,0 +1,560 @@ +import invariant from "invariant"; +import env from "@server/env"; +import Logger from "@server/logging/Logger"; +import { + Collection, + FileOperation, + Group, + Integration, + Pin, + Star, + Team, + WebhookDelivery, + WebhookSubscription, + Document, + User, + Revision, + View, + Share, + CollectionUser, + CollectionGroup, + GroupUser, +} from "@server/models"; +import { + presentCollection, + presentDocument, + presentRevision, + presentFileOperation, + presentGroup, + presentIntegration, + presentPin, + presentStar, + presentTeam, + presentUser, + presentWebhook, + presentWebhookSubscription, + presentView, + presentShare, + presentMembership, + presentGroupMembership, + presentCollectionGroupMembership, +} from "@server/presenters"; +import { WebhookPayload } from "@server/presenters/webhook"; +import { + CollectionEvent, + CollectionGroupEvent, + CollectionUserEvent, + DocumentEvent, + Event, + FileOperationEvent, + GroupEvent, + GroupUserEvent, + IntegrationEvent, + PinEvent, + RevisionEvent, + ShareEvent, + StarEvent, + TeamEvent, + UserEvent, + ViewEvent, + WebhookSubscriptionEvent, +} from "@server/types"; +import BaseTask from "./BaseTask"; + +function assertUnreachable(event: never) { + Logger.warn(`DeliverWebhookTask did not handle ${(event as any).name}`); +} + +type Props = { + subscriptionId: string; + event: Event; +}; + +export default class DeliverWebhookTask extends BaseTask { + public async perform({ subscriptionId, event }: Props) { + const subscription = await WebhookSubscription.findByPk(subscriptionId); + invariant(subscription, "Subscription not found"); + + Logger.info( + "task", + `DeliverWebhookTask: ${event.name} for ${subscription.name}` + ); + + switch (event.name) { + case "api_keys.create": + case "api_keys.delete": + // Ignored + return; + case "users.create": + case "users.signin": + case "users.signout": + case "users.update": + case "users.suspend": + case "users.activate": + case "users.delete": + case "users.invite": + await this.handleUserEvent(subscription, event); + return; + case "documents.create": + case "documents.publish": + case "documents.unpublish": + case "documents.delete": + case "documents.permanent_delete": + case "documents.archive": + case "documents.unarchive": + case "documents.restore": + case "documents.star": + case "documents.unstar": + case "documents.move": + case "documents.update": + case "documents.title_change": + await this.handleDocumentEvent(subscription, event); + return; + case "documents.update.delayed": + case "documents.update.debounced": + // Ignored + return; + case "revisions.create": + await this.handleRevisionEvent(subscription, event); + return; + case "fileOperations.create": + case "fileOperations.update": + case "fileOperation.delete": + await this.handleFileOperationEvent(subscription, event); + return; + case "collections.create": + case "collections.update": + case "collections.delete": + case "collections.move": + case "collections.permission_changed": + await this.handleCollectionEvent(subscription, event); + return; + case "collections.add_user": + case "collections.remove_user": + await this.handleCollectionUserEvent(subscription, event); + return; + case "collections.add_group": + case "collections.remove_group": + await this.handleCollectionGroupEvent(subscription, event); + return; + case "groups.create": + case "groups.update": + case "groups.delete": + await this.handleGroupEvent(subscription, event); + return; + case "groups.add_user": + case "groups.remove_user": + await this.handleGroupUserEvent(subscription, event); + return; + case "integrations.create": + case "integrations.update": + await this.handleIntegrationEvent(subscription, event); + return; + case "teams.update": + await this.handleTeamEvent(subscription, event); + return; + case "pins.create": + case "pins.update": + case "pins.delete": + await this.handlePinEvent(subscription, event); + return; + case "stars.create": + case "stars.update": + case "stars.delete": + await this.handleStarEvent(subscription, event); + return; + case "shares.create": + case "shares.update": + case "shares.revoke": + await this.handleShareEvent(subscription, event); + return; + case "webhook_subscriptions.create": + case "webhook_subscriptions.delete": + case "webhook_subscriptions.update": + await this.handleWebhookSubscriptionEvent(subscription, event); + return; + case "views.create": + await this.handleViewEvent(subscription, event); + return; + default: + assertUnreachable(event); + } + } + + private async handleWebhookSubscriptionEvent( + subscription: WebhookSubscription, + event: WebhookSubscriptionEvent + ): Promise { + const model = await WebhookSubscription.findByPk(event.modelId, { + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: event.modelId, + model: model && presentWebhookSubscription(model), + }, + }); + } + + private async handleViewEvent( + subscription: WebhookSubscription, + event: ViewEvent + ): Promise { + const model = await View.scope("withUser").findByPk(event.modelId, { + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: event.modelId, + model: model && presentView(model), + }, + }); + } + + private async handleStarEvent( + subscription: WebhookSubscription, + event: StarEvent + ): Promise { + const model = await Star.findByPk(event.modelId, { + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: event.modelId, + model: model && presentStar(model), + }, + }); + } + + private async handleShareEvent( + subscription: WebhookSubscription, + event: ShareEvent + ): Promise { + const model = await Share.findByPk(event.modelId, { + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: event.modelId, + model: model && presentShare(model), + }, + }); + } + + private async handlePinEvent( + subscription: WebhookSubscription, + event: PinEvent + ): Promise { + const model = await Pin.findByPk(event.modelId, { + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: event.modelId, + model: model && presentPin(model), + }, + }); + } + + private async handleTeamEvent( + subscription: WebhookSubscription, + event: TeamEvent + ): Promise { + const model = await Team.scope("withDomains").findByPk(event.teamId, { + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: event.teamId, + model: model && presentTeam(model), + }, + }); + } + + private async handleIntegrationEvent( + subscription: WebhookSubscription, + event: IntegrationEvent + ): Promise { + const model = await Integration.findByPk(event.modelId, { + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: event.modelId, + model: model && presentIntegration(model), + }, + }); + } + + private async handleGroupEvent( + subscription: WebhookSubscription, + event: GroupEvent + ): Promise { + const model = await Group.findByPk(event.modelId, { + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: event.modelId, + model: model && presentGroup(model), + }, + }); + } + + private async handleGroupUserEvent( + subscription: WebhookSubscription, + event: GroupUserEvent + ): Promise { + const model = await GroupUser.scope(["withUser", "withGroup"]).findOne({ + where: { + groupId: event.modelId, + userId: event.userId, + }, + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: `${event.userId}-${event.modelId}`, + model: model && presentGroupMembership(model), + group: model && presentGroup(model.group), + user: model && presentUser(model.user), + }, + }); + } + + private async handleCollectionEvent( + subscription: WebhookSubscription, + event: CollectionEvent + ): Promise { + const model = await Collection.findByPk(event.collectionId, { + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: event.collectionId, + model: model && presentCollection(model), + }, + }); + } + + private async handleCollectionUserEvent( + subscription: WebhookSubscription, + event: CollectionUserEvent + ): Promise { + const model = await CollectionUser.scope([ + "withUser", + "withCollection", + ]).findOne({ + where: { + collectionId: event.collectionId, + userId: event.userId, + }, + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: `${event.userId}-${event.collectionId}`, + model: model && presentMembership(model), + collection: model && presentCollection(model.collection), + user: model && presentUser(model.user), + }, + }); + } + + private async handleCollectionGroupEvent( + subscription: WebhookSubscription, + event: CollectionGroupEvent + ): Promise { + const model = await CollectionGroup.scope([ + "withGroup", + "withCollection", + ]).findOne({ + where: { + collectionId: event.collectionId, + groupId: event.modelId, + }, + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: `${event.modelId}-${event.collectionId}`, + model: model && presentCollectionGroupMembership(model), + collection: model && presentCollection(model.collection), + group: model && presentGroup(model.group), + }, + }); + } + + private async handleFileOperationEvent( + subscription: WebhookSubscription, + event: FileOperationEvent + ): Promise { + const model = await FileOperation.findByPk(event.modelId, { + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: event.modelId, + model: model && presentFileOperation(model), + }, + }); + } + + private async handleDocumentEvent( + subscription: WebhookSubscription, + event: DocumentEvent + ): Promise { + const model = await Document.findByPk(event.documentId, { + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: event.documentId, + model: model && (await presentDocument(model)), + }, + }); + } + + private async handleRevisionEvent( + subscription: WebhookSubscription, + event: RevisionEvent + ): Promise { + const model = await Revision.findByPk(event.modelId, { + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: event.modelId, + model: model && (await presentRevision(model)), + }, + }); + } + + private async handleUserEvent( + subscription: WebhookSubscription, + event: UserEvent + ): Promise { + const model = await User.findByPk(event.userId, { + paranoid: false, + }); + + await this.sendWebhook({ + event, + subscription, + payload: { + id: event.userId, + model: model && presentUser(model), + }, + }); + } + + private async sendWebhook({ + event, + subscription, + payload, + }: { + event: Event; + subscription: WebhookSubscription; + payload: WebhookPayload; + }) { + const delivery = await WebhookDelivery.create({ + webhookSubscriptionId: subscription.id, + status: "pending", + }); + + let response, requestBody, requestHeaders, status; + try { + requestBody = presentWebhook({ + event, + delivery, + payload, + }); + requestHeaders = { + "Content-Type": "application/json", + "user-agent": `Outline-Webhooks${env.VERSION ? `/${env.VERSION}` : ""}`, + }; + response = await fetch(subscription.url, { + method: "POST", + headers: requestHeaders, + body: JSON.stringify(requestBody), + }); + status = response.ok ? "success" : "failed"; + } catch (err) { + status = "failed"; + } + + await delivery.update({ + status, + statusCode: response ? response.status : null, + requestBody, + requestHeaders, + responseBody: response ? await response.text() : "", + responseHeaders: response + ? Object.fromEntries(response.headers.entries()) + : {}, + }); + + if (response && !response.ok) { + const recentDeliveries = await WebhookDelivery.findAll({ + where: { + webhookSubscriptionId: subscription.id, + }, + order: [["createdAt", "DESC"]], + limit: 25, + }); + + const allFailed = recentDeliveries.every( + (delivery) => delivery.status === "failed" + ); + + if (recentDeliveries.length === 25 && allFailed) { + await subscription.update({ enabled: false }); + } + } + } +} diff --git a/server/routes/api/auth.ts b/server/routes/api/auth.ts index 93f0b2237..75077c8c9 100644 --- a/server/routes/api/auth.ts +++ b/server/routes/api/auth.ts @@ -5,7 +5,7 @@ import { parseDomain } from "@shared/utils/domains"; import { sequelize } from "@server/database/sequelize"; import env from "@server/env"; import auth from "@server/middlewares/authentication"; -import { Event, Team, TeamDomain } from "@server/models"; +import { Event, Team } from "@server/models"; import { presentUser, presentTeam, presentPolicies } from "@server/presenters"; import ValidateSSOAccessTask from "@server/queues/tasks/ValidateSSOAccessTask"; import providers from "../auth/providers"; @@ -108,9 +108,7 @@ router.post("auth.config", async (ctx) => { router.post("auth.info", auth(), async (ctx) => { const { user } = ctx.state; - const team = await Team.findByPk(user.teamId, { - include: [{ model: TeamDomain }], - }); + const team = await Team.scope("withDomains").findByPk(user.teamId); invariant(team, "Team not found"); await ValidateSSOAccessTask.schedule({ userId: user.id }); diff --git a/server/routes/api/cron.ts b/server/routes/api/cron.ts index b6ee646db..340f5e72d 100644 --- a/server/routes/api/cron.ts +++ b/server/routes/api/cron.ts @@ -5,6 +5,7 @@ import { AuthenticationError } from "@server/errors"; import CleanupDeletedDocumentsTask from "@server/queues/tasks/CleanupDeletedDocumentsTask"; import CleanupDeletedTeamsTask from "@server/queues/tasks/CleanupDeletedTeamsTask"; import CleanupExpiredFileOperationsTask from "@server/queues/tasks/CleanupExpiredFileOperationsTask"; +import CleanupWebhookDeliveriesTask from "@server/queues/tasks/CleanupWebhookDeliveriesTask"; import InviteReminderTask from "@server/queues/tasks/InviteReminderTask"; const router = new Router(); @@ -22,6 +23,8 @@ const cronHandler = async (ctx: Context) => { await CleanupDeletedTeamsTask.schedule({ limit }); + await CleanupWebhookDeliveriesTask.schedule({ limit }); + await InviteReminderTask.schedule(); ctx.body = { diff --git a/server/routes/api/groups.ts b/server/routes/api/groups.ts index d6495b89e..8634f814e 100644 --- a/server/routes/api/groups.ts +++ b/server/routes/api/groups.ts @@ -45,7 +45,9 @@ router.post("groups.list", auth(), pagination(), async (ctx) => { .slice(0, MAX_AVATAR_DISPLAY) ) .flat() - .map(presentGroupMembership), + .map((membership) => + presentGroupMembership(membership, { includeUser: true }) + ), }, policies: presentPolicies(user, groups), }; @@ -191,7 +193,9 @@ router.post("groups.memberships", auth(), pagination(), async (ctx) => { ctx.body = { pagination: ctx.state.pagination, data: { - groupMemberships: memberships.map(presentGroupMembership), + groupMemberships: memberships.map((membership) => + presentGroupMembership(membership, { includeUser: true }) + ), users: memberships.map((membership) => presentUser(membership.user)), }, }; @@ -250,7 +254,9 @@ router.post("groups.add_user", auth(), async (ctx) => { ctx.body = { data: { users: [presentUser(user)], - groupMemberships: [presentGroupMembership(membership)], + groupMemberships: [ + presentGroupMembership(membership, { includeUser: true }), + ], groups: [presentGroup(group)], }, }; diff --git a/server/routes/api/index.ts b/server/routes/api/index.ts index 9ee309149..180af9da0 100644 --- a/server/routes/api/index.ts +++ b/server/routes/api/index.ts @@ -27,6 +27,7 @@ import stars from "./stars"; import team from "./team"; import users from "./users"; import views from "./views"; +import webhookSubscriptions from "./webhookSubscriptions"; const api = new Koa(); const router = new Router(); @@ -67,6 +68,7 @@ router.use("/", attachments.routes()); router.use("/", utils.routes()); router.use("/", groups.routes()); router.use("/", fileOperationsRoute.routes()); +router.use("/", webhookSubscriptions.routes()); router.post("*", (ctx) => { ctx.throw(NotFoundError("Endpoint not found")); diff --git a/server/routes/api/views.ts b/server/routes/api/views.ts index 59a2d5e4b..e323d34a2 100644 --- a/server/routes/api/views.ts +++ b/server/routes/api/views.ts @@ -44,6 +44,7 @@ router.post("views.create", auth(), async (ctx) => { documentId: document.id, collectionId: document.collectionId, teamId: user.teamId, + modelId: view.id, data: { title: document.title, }, diff --git a/server/routes/api/webhookSubscriptions.ts b/server/routes/api/webhookSubscriptions.ts new file mode 100644 index 000000000..085031a9e --- /dev/null +++ b/server/routes/api/webhookSubscriptions.ts @@ -0,0 +1,137 @@ +import Router from "koa-router"; +import { compact } from "lodash"; +import { ValidationError } from "@server/errors"; +import auth from "@server/middlewares/authentication"; +import { WebhookSubscription, Event } from "@server/models"; +import { authorize } from "@server/policies"; +import { presentWebhookSubscription } from "@server/presenters"; +import { WebhookSubscriptionEvent } from "@server/types"; +import { assertArray, assertPresent, assertUuid } from "@server/validation"; +import pagination from "./middlewares/pagination"; + +const router = new Router(); + +router.post("webhookSubscriptions.list", auth(), pagination(), async (ctx) => { + const { user } = ctx.state; + authorize(user, "listWebhookSubscription", user.team); + const webhooks = await WebhookSubscription.findAll({ + where: { + teamId: user.teamId, + }, + order: [["createdAt", "DESC"]], + offset: ctx.state.pagination.offset, + limit: ctx.state.pagination.limit, + }); + + ctx.body = { + pagination: ctx.state.pagination, + data: webhooks.map(presentWebhookSubscription), + }; +}); + +router.post("webhookSubscriptions.create", auth(), async (ctx) => { + const { user } = ctx.state; + authorize(user, "createWebhookSubscription", user.team); + + const { name, url } = ctx.request.body; + const events: string[] = compact(ctx.request.body.events); + assertPresent(name, "name is required"); + assertPresent(url, "url is required"); + assertArray(events, "events is required"); + if (events.length === 0) { + throw ValidationError("events are required"); + } + + const webhookSubscription = await WebhookSubscription.create({ + name, + events, + createdById: user.id, + teamId: user.teamId, + url, + enabled: true, + }); + + const event: WebhookSubscriptionEvent = { + name: "webhook_subscriptions.create", + modelId: webhookSubscription.id, + teamId: user.teamId, + actorId: user.id, + data: { + name, + url, + events, + }, + ip: ctx.request.ip, + }; + await Event.create(event); + + ctx.body = { + data: presentWebhookSubscription(webhookSubscription), + }; +}); + +router.post("webhookSubscriptions.delete", auth(), async (ctx) => { + const { id } = ctx.body; + assertUuid(id, "id is required"); + const { user } = ctx.state; + const webhookSubscription = await WebhookSubscription.findByPk(id); + + authorize(user, "delete", webhookSubscription); + + await webhookSubscription.destroy(); + + const event: WebhookSubscriptionEvent = { + name: "webhook_subscriptions.delete", + modelId: webhookSubscription.id, + teamId: user.teamId, + actorId: user.id, + data: { + name: webhookSubscription.name, + url: webhookSubscription.url, + events: webhookSubscription.events, + }, + ip: ctx.request.ip, + }; + await Event.create(event); +}); + +router.post("webhookSubscriptions.update", auth(), async (ctx) => { + const { id } = ctx.body; + assertUuid(id, "id is required"); + const { user } = ctx.state; + + const { name, url } = ctx.request.body; + const events: string[] = compact(ctx.request.body.events); + assertPresent(name, "name is required"); + assertPresent(url, "url is required"); + assertArray(events, "events is required"); + if (events.length === 0) { + throw ValidationError("events are required"); + } + + const webhookSubscription = await WebhookSubscription.findByPk(id); + + authorize(user, "update", webhookSubscription); + + await webhookSubscription.update({ name, url, events, enabled: true }); + + const event: WebhookSubscriptionEvent = { + name: "webhook_subscriptions.update", + modelId: webhookSubscription.id, + teamId: user.teamId, + actorId: user.id, + data: { + name: webhookSubscription.name, + url: webhookSubscription.url, + events: webhookSubscription.events, + }, + ip: ctx.request.ip, + }; + await Event.create(event); + + ctx.body = { + data: presentWebhookSubscription(webhookSubscription), + }; +}); + +export default router; diff --git a/server/test/factories.ts b/server/test/factories.ts index ba3cde030..d45edd37c 100644 --- a/server/test/factories.ts +++ b/server/test/factories.ts @@ -14,6 +14,8 @@ import { Integration, AuthenticationProvider, FileOperation, + WebhookSubscription, + WebhookDelivery, } from "@server/models"; import { FileOperationState, @@ -366,3 +368,58 @@ export async function buildAttachment(overrides: Partial = {}) { ...overrides, }); } + +export async function buildWebhookSubscription( + overrides: Partial = {} +): Promise { + if (!overrides.teamId) { + const team = await buildTeam(); + overrides.teamId = team.id; + } + if (!overrides.createdById) { + const user = await buildUser({ + teamId: overrides.teamId, + }); + overrides.createdById = user.id; + } + if (!overrides.name) { + overrides.name = "Test Webhook Subscription"; + } + if (!overrides.url) { + overrides.url = "https://www.example.com/webhook"; + } + if (!overrides.events) { + overrides.events = ["*"]; + } + if (!overrides.enabled) { + overrides.enabled = true; + } + + return WebhookSubscription.create(overrides); +} + +export async function buildWebhookDelivery( + overrides: Partial = {} +): Promise { + if (!overrides.status) { + overrides.status = "success"; + } + if (!overrides.statusCode) { + overrides.statusCode = 200; + } + if (!overrides.requestBody) { + overrides.requestBody = "{}"; + } + if (!overrides.requestHeaders) { + overrides.requestHeaders = {}; + } + if (!overrides.webhookSubscriptionId) { + const webhookSubscription = await buildWebhookSubscription(); + overrides.webhookSubscriptionId = webhookSubscription.id; + } + if (!overrides.createdAt) { + overrides.createdAt = new Date(); + } + + return WebhookDelivery.create(overrides); +} diff --git a/server/types.ts b/server/types.ts index 8c387974e..d3acfe029 100644 --- a/server/types.ts +++ b/server/types.ts @@ -1,5 +1,5 @@ import { Context } from "koa"; -import { FileOperation, User } from "./models"; +import { FileOperation, Team, User } from "./models"; export type ContextWithState = Context & { state: { @@ -9,227 +9,241 @@ export type ContextWithState = Context & { }; }; -export type UserEvent = - | { - name: "users.create" // eslint-disable-line - | "users.signin" - | "users.signout" - | "users.update" - | "users.suspend" - | "users.activate" - | "users.delete"; - userId: string; - teamId: string; - actorId: string; - ip: string; - } - | { - name: "users.invite"; - teamId: string; - actorId: string; - data: { - email: string; - name: string; - }; - ip: string; - }; +type BaseEvent = { + teamId: string; + actorId: string; + ip: string; +}; -export type DocumentEvent = - | { - name: "documents.create" // eslint-disable-line - | "documents.publish" - | "documents.unpublish" - | "documents.delete" - | "documents.permanent_delete" - | "documents.archive" - | "documents.unarchive" - | "documents.restore" - | "documents.star" - | "documents.unstar"; - documentId: string; - collectionId: string; - teamId: string; - actorId: string; - ip: string; - data: { - title: string; - source?: "import"; - }; - } - | { - name: "documents.move"; - documentId: string; - collectionId: string; - teamId: string; - actorId: string; - data: { - collectionIds: string[]; - documentIds: string[]; - }; - ip: string; - } - | { - name: "documents.update" // eslint-disable-line - | "documents.update.delayed" - | "documents.update.debounced"; - documentId: string; - collectionId: string; - createdAt: string; - teamId: string; - actorId: string; - data: { - title: string; - autosave: boolean; - done: boolean; - }; - ip: string; - } - | { - name: "documents.title_change"; - documentId: string; - collectionId: string; - createdAt: string; - teamId: string; - actorId: string; - data: { - title: string; - previousTitle: string; - }; - ip: string; - }; +export type ApiKeyEvent = BaseEvent & { + name: "api_keys.create" | "api_keys.delete"; + modelId: string; + data: { + name: string; + }; +}; -export type RevisionEvent = { +export type UserEvent = BaseEvent & + ( + | { + name: + | "users.create" + | "users.signin" + | "users.signout" + | "users.update" + | "users.suspend" + | "users.activate" + | "users.delete"; + userId: string; + } + | { + name: "users.invite"; + userId: string; + data: { + email: string; + name: string; + }; + } + ); + +export type DocumentEvent = BaseEvent & + ( + | { + name: + | "documents.create" + | "documents.publish" + | "documents.unpublish" + | "documents.delete" + | "documents.permanent_delete" + | "documents.archive" + | "documents.unarchive" + | "documents.restore" + | "documents.star" + | "documents.unstar"; + documentId: string; + collectionId: string; + data: { + title: string; + source?: "import"; + }; + } + | { + name: "documents.move"; + documentId: string; + collectionId: string; + data: { + collectionIds: string[]; + documentIds: string[]; + }; + } + | { + name: + | "documents.update" + | "documents.update.delayed" + | "documents.update.debounced"; + documentId: string; + collectionId: string; + createdAt: string; + data: { + title: string; + autosave: boolean; + done: boolean; + }; + } + | { + name: "documents.title_change"; + documentId: string; + collectionId: string; + createdAt: string; + data: { + title: string; + previousTitle: string; + }; + } + ); + +export type RevisionEvent = BaseEvent & { name: "revisions.create"; documentId: string; collectionId: string; - teamId: string; + modelId: string; }; -export type FileOperationEvent = { +export type FileOperationEvent = BaseEvent & { name: | "fileOperations.create" | "fileOperations.update" | "fileOperation.delete"; - teamId: string; - actorId: string; modelId: string; data: Partial; }; -export type CollectionEvent = - | { - name: "collections.create" // eslint-disable-line - | "collections.update" - | "collections.delete"; - collectionId: string; - teamId: string; - actorId: string; - data: { - name: string; - }; - ip: string; - } - | { - name: "collections.add_user" | "collections.remove_user"; - userId: string; - collectionId: string; - teamId: string; - actorId: string; - ip: string; - } - | { - name: "collections.add_group" | "collections.remove_group"; - collectionId: string; - teamId: string; - actorId: string; - modelId: string; - data: { - name: string; - }; - ip: string; - } - | { - name: "collections.move"; - collectionId: string; - teamId: string; - actorId: string; - data: { - index: string; - }; - ip: string; - } - | { - name: "collections.permission_changed"; - collectionId: string; - teamId: string; - actorId: string; - data: { - privacyChanged: boolean; - sharingChanged: boolean; - }; - ip: string; - }; +export type CollectionUserEvent = BaseEvent & { + name: "collections.add_user" | "collections.remove_user"; + userId: string; + collectionId: string; +}; -export type GroupEvent = - | { - name: "groups.create" | "groups.delete" | "groups.update"; - actorId: string; - modelId: string; - teamId: string; - data: { - name: string; - }; - ip: string; - } - | { - name: "groups.add_user" | "groups.remove_user"; - actorId: string; - userId: string; - modelId: string; - teamId: string; - data: { - name: string; - }; - ip: string; - }; +export type CollectionGroupEvent = BaseEvent & { + name: "collections.add_group" | "collections.remove_group"; + collectionId: string; + modelId: string; + data: { + name: string; + }; +}; -export type IntegrationEvent = { +export type CollectionEvent = BaseEvent & + ( + | CollectionUserEvent + | CollectionGroupEvent + | { + name: + | "collections.create" + | "collections.update" + | "collections.delete"; + collectionId: string; + data: { + name: string; + }; + } + | { + name: "collections.move"; + collectionId: string; + data: { + index: string; + }; + } + | { + name: "collections.permission_changed"; + collectionId: string; + data: { + privacyChanged: boolean; + sharingChanged: boolean; + }; + } + ); + +export type GroupUserEvent = BaseEvent & { + name: "groups.add_user" | "groups.remove_user"; + userId: string; + modelId: string; + data: { + name: string; + }; +}; + +export type GroupEvent = BaseEvent & + ( + | GroupUserEvent + | { + name: "groups.create" | "groups.delete" | "groups.update"; + modelId: string; + data: { + name: string; + }; + } + ); + +export type IntegrationEvent = BaseEvent & { name: "integrations.create" | "integrations.update"; modelId: string; - teamId: string; - actorId: string; - ip: string; }; -export type TeamEvent = { +export type TeamEvent = BaseEvent & { name: "teams.update"; - teamId: string; - actorId: string; - data: Record; - ip: string; + data: Partial; }; -export type PinEvent = { +export type PinEvent = BaseEvent & { name: "pins.create" | "pins.update" | "pins.delete"; - teamId: string; modelId: string; documentId: string; collectionId?: string; - actorId: string; - ip: string; }; -export type StarEvent = { +export type StarEvent = BaseEvent & { name: "stars.create" | "stars.update" | "stars.delete"; - teamId: string; modelId: string; documentId: string; userId: string; - actorId: string; - ip: string; +}; + +export type ShareEvent = BaseEvent & { + name: "shares.create" | "shares.update" | "shares.revoke"; + modelId: string; + documentId: string; + collectionId?: string; + data: { + name: string; + }; +}; + +export type ViewEvent = BaseEvent & { + name: "views.create"; + documentId: string; + collectionId: string; + modelId: string; + data: { + title: string; + }; +}; + +export type WebhookSubscriptionEvent = BaseEvent & { + name: + | "webhook_subscriptions.create" + | "webhook_subscriptions.delete" + | "webhook_subscriptions.update"; + modelId: string; + data: { + name: string; + url: string; + events: string[]; + }; }; export type Event = + | ApiKeyEvent | UserEvent | DocumentEvent | PinEvent @@ -239,4 +253,7 @@ export type Event = | IntegrationEvent | GroupEvent | RevisionEvent - | TeamEvent; + | ShareEvent + | TeamEvent + | ViewEvent + | WebhookSubscriptionEvent; diff --git a/shared/i18n/locales/en_US/translation.json b/shared/i18n/locales/en_US/translation.json index 2f45e4d98..721dfe21a 100644 --- a/shared/i18n/locales/en_US/translation.json +++ b/shared/i18n/locales/en_US/translation.json @@ -186,6 +186,7 @@ "Share Links": "Share Links", "Import": "Import", "Export": "Export", + "Webhooks": "Webhooks", "Integrations": "Integrations", "Insert column after": "Insert column after", "Insert column before": "Insert column before", @@ -586,6 +587,20 @@ "Active": "Active", "Everyone": "Everyone", "Admins": "Admins", + "Are you sure you want to delete the {{ name }} webhook?": "Are you sure you want to delete the {{ name }} webhook?", + "Webhook updated": "Webhook updated", + "Update": "Update", + "Updating": "Updating", + "Provide a descriptive name for this webhook and the URL we should send a POST request to when matching events are created.": "Provide a descriptive name for this webhook and the URL we should send a POST request to when matching events are created.", + "Subscribe to all events, groups, or individual events. We recommend only subscribing to the minimum amount of events that your application needs to function.": "Subscribe to all events, groups, or individual events. We recommend only subscribing to the minimum amount of events that your application needs to function.", + "URL": "URL", + "All events": "All events", + "All {{ groupName }} events": "All {{ groupName }} events", + "Delete webhook": "Delete webhook", + "Disabled": "Disabled", + "Subscribed events": "Subscribed events", + "Edit webhook": "Edit webhook", + "Webhook created": "Webhook created", "Logo updated": "Logo updated", "Unable to upload new logo": "Unable to upload new logo", "These settings affect the way that your knowledge base appears to everyone on the team.": "These settings affect the way that your knowledge base appears to everyone on the team.", @@ -682,6 +697,9 @@ "You can create an unlimited amount of personal tokens to authenticate\n with the API. Tokens have the same permissions as your user account.\n For more details see the developer documentation.": "You can create an unlimited amount of personal tokens to authenticate\n with the API. Tokens have the same permissions as your user account.\n For more details see the developer documentation.", "Tokens": "Tokens", "Create a token": "Create a token", + "New webhook": "New webhook", + "Webhooks can be used to notify your application when events happen in Outline. Events are sent as a https request with a JSON payload in near real-time.": "Webhooks can be used to notify your application when events happen in Outline. Events are sent as a https request with a JSON payload in near real-time.", + "Create a webhook": "Create a webhook", "Zapier is a platform that allows Outline to easily integrate with thousands of other business tools. Head over to Zapier to setup a \"Zap\" and start programmatically interacting with Outline.'": "Zapier is a platform that allows Outline to easily integrate with thousands of other business tools. Head over to Zapier to setup a \"Zap\" and start programmatically interacting with Outline.'", "Open Zapier": "Open Zapier", "Alphabetical": "Alphabetical", diff --git a/yarn.lock b/yarn.lock index 2e6e2eb66..98ff4f04d 100644 --- a/yarn.lock +++ b/yarn.lock @@ -11330,10 +11330,10 @@ os-browserify@^0.3.0: resolved "https://registry.yarnpkg.com/os-browserify/-/os-browserify-0.3.0.tgz#854373c7f5c2315914fc9bfc6bd8238fdda1ec27" integrity sha1-hUNzx/XCMVkU/Jv8a9gjj92h7Cc= -outline-icons@^1.42.0: - version "1.42.0" - resolved "https://registry.yarnpkg.com/outline-icons/-/outline-icons-1.42.0.tgz#76b02f57b9dcac79c1f6876e918d472fefaa4568" - integrity sha512-px2wNGrzTDCU0pxUO2pKs5dgE3fdz3bQ3lpLA9CgzBpkcadpOQgRxxxyBQ12fyOXyKC0L4evAJeomoAtvMWISQ== +outline-icons@^1.43.1: + version "1.43.1" + resolved "https://registry.yarnpkg.com/outline-icons/-/outline-icons-1.43.1.tgz#3193c4c659c66b34788db043bb2f843b9c437a48" + integrity sha512-REj+JsCFi2Jv5uG0/OrBsMVSBFAIsSROxynWbuO9r2eNT8wdqjni02Mk1gq1qFfTbwOvHJ+7ycadu6zlISAK2g== oy-vey@^0.10.0: version "0.10.0" @@ -12483,6 +12483,11 @@ react-helmet@^6.1.0: react-fast-compare "^3.1.1" react-side-effect "^2.1.0" +react-hook-form@^7.31.2: + version "7.31.2" + resolved "https://registry.yarnpkg.com/react-hook-form/-/react-hook-form-7.31.2.tgz#efb7ac469810954488b7cf40be4e5017122c6e5e" + integrity sha512-oPudn3YuyzWg//IsT9z2cMEjWocAgHWX/bmueDT8cmsYQnGY5h7/njjvMDfLVv3mbdhYBjslTRnII2MIT7eNCA== + react-i18next@^11.16.6: version "11.16.6" resolved "https://registry.yarnpkg.com/react-i18next/-/react-i18next-11.16.6.tgz#e8a07802c391a55e1528673201a2727994787641"