204 Commits

Author SHA1 Message Date
Tom Moor
7ae892fe06 fix: Long collection description prevents import (#3847)
* fix: Long collection description prevents import
fix: Parallelize attachment upload during import

* fix: Improve Notion image import matching

* chore: Bump JSZIP (perf)

* fix: Allow redirect from /doc/<id> to canonical url

* fix: Importing document with only title duplicates title in body
2022-07-24 01:37:20 -07:00
Tom Moor
4f537c7578 Remove retry on export task 2022-07-23 17:00:32 +01:00
Tom Moor
dc94a683e7 chore: Reduce timeout on webhook deliveries 2022-07-17 18:48:45 +01:00
Tom Moor
47e73cee4e feat: Cleanup api keys and webhooks for suspended users (#3756) 2022-07-13 00:59:31 -07:00
Tom Moor
c6fdffba77 chore: Internal request filtering 2022-07-05 11:06:47 +02:00
Tom Moor
ee10e1407a fix: Typo of fileOperation -> fileOperations 2022-07-03 16:27:03 +02:00
Tom Moor
62d9bf7105 chore: Move initial avatar upload to background worker (#3727)
* chore: Async user avatar upload processor

* chore: Async team avatar upload

* Refactor to task for retries

* Docs
Include avatarUrl in task props to prevent race condition
Remove transaction around upload fetch request
2022-07-03 02:36:15 -07:00
Tom Moor
ee22a127f6 feat: Add email when webhook is disabled (#3721)
fix: Webhook not disabled under some error conditions
2022-07-02 05:36:40 -07:00
Tom Moor
c9cd424a8d chore: Remove over-usage of invariant (#3719) 2022-07-02 05:29:39 -07:00
Tom Moor
108b5b934a fix: users.promote & users.demote not handled by DeliverWebhookTask 2022-07-02 14:24:49 +02:00
Tom Moor
94824af6e7 fix: Allow soft-deleted records to be queried from RevisionProcessor
closes #3706
2022-07-02 11:58:22 +02:00
Tom Moor
7ce57c9c83 fix: attachments events not recognised by DeliverWebhookTask 2022-07-01 18:40:32 +02:00
Tom Moor
b44dc726f3 test: fix fetch related tests 2022-06-30 10:37:06 +02:00
Tom Moor
930bfd5391 fix: Must import fetch, log errors, use git short sha for version 2022-06-29 08:28:44 +02:00
Tom Moor
10f86ed218 feat: Webhooks (#3691)
* Webhooks (#3607)

* Get the migration and the model setup. Also make the sample env file a bit easier to use. Now just requires setting a SECRET_KEY and besides that will boot up from the sample

* WIP: Start getting a Webhook page created. Just the skeleton state right now

* WIP: Getting a form created to create webhooks, need to bring in react-hook-forms now

* WIP: Get library installed and make TS happy

* Get a few checkboxes ready to go

* Get creating and destroying working with a decent start to a frontend

* Didn't mean to enable this

* Remove eslint and fix other random typescript issue

* Rename some events to be more realistic

* Revert these changes

* PR review comments around policies. Also make sure this inherits from IdModel so it actually gets an id

* Allow any admin on the team to edit webhooks

* Start sending some webhooks for some User events

* Make sure the URL is valid

* Start recording webhook deliveries

* Make sure to verify if the subscription is for the type of event we are looking at

* Refactor sending Webhooks and follow better webhook schema

This creates a presenter to unify the format of webhooks. We also
extract the sending of webhooks and recording their deliveries to a
method than can be used by each of the different event type methods

We also add a status to WebhookDelivery since we need to save the record
before we make the HTTP request to get its id. Then once we make the
request and get a response we can update the delivery with the HTTP info

* Turn off a subscription that has failed for the last 25 deliveries

* Get a first spec passing. Found a bug in my returning of promises so good to patch that up now

* This looks nicer

* Get some tests added for the processor

* Add cron task to delete older webhooks

* Add Document Events to the Processor

* Revisions, FileOperations and Collections

* Get all the server side events added to the processor and make Typescript make sure they are all accounted for

* Get all the events added to the Frontend and work on styling them a bit, still needs some love though

* Get UI styled up a bit

* Get events wired up for webhook subscriptions

* Get delete events working and test at least one variant of them

* Get deletes working and actually make sure to send the model id in the webhook

* Remove webhook secrets from this slice

* Add disabled label for subscriptions that are disabled

* Make sure to cascade the delete

* Reorg this file a bit

* Fix association

* I removed secret for the moment

* Apply Copy changes from PR Review

Co-authored-by: Tom Moor <tom.moor@gmail.com>

* Actually apply the copy changes

TIL that if you Resolve a conversation it _also_ removes the 'staged suggestion' from your list on Github

Co-authored-by: Tom Moor <tom.moor@gmail.com>

* Update app/scenes/Settings/Webhooks.tsx

Missed this copy change before

Co-authored-by: Tom Moor <tom.moor@gmail.com>

* Add disabled as yellow badge

* Resolve frontend comments

* Fixup Schema a bit and remove the dependency on the subscription

* Add test to make sure we don't disable until there are enough failures, and fix code to actually do that. Also some test fixes from the json response shape changes

* Fix WebhookDeliveries to store the responses as Text instead of blobs

* Switch to text better for response bodies, this is using the helpers better and makes the code read better

* Move the logic to a task but run in through the processor cause the tests expect that right now, moving the tests over next

* Split up the tests and actually enqueue the events from the WebhookProcessor instead of doing them inline

* Allow any team admin to see any webhook subscription for the team

* Add the indexes based on our lookup patterns

* Run eslint --fix to fix auto correct issues from when I tried to use Github to merge copy changes

* Allow subscriptions to be edited after creation

* Types caught that I didn't add the new event to the webhook processor, also added it to the frontend here

* I think this will get these into the translations file

* Catch a few more translations, use styled components better and remove usage of webhook subscription in the copy

Co-authored-by: Tom Moor <tom.moor@gmail.com>

* fix: tsc
fix: Document model payload empty

* fix: Revision webhook payload
Add custom UA for hooks

* Add webhooks icon, move under Integrations settings
Some spacing fixes

* Add actorId to webhook payloads

* Add View and ApiKey event types

* Spacing tweaks, fix team payload

* fix: Webhook not disabled after 25 failures

* fix: Enable webhook when editing if previously disabled

* fix: Correctly store response headers

* fix: Error in json/parsing/presentation results in hanging 'pending' webhook delivery

* fix: Awkward payload for users.invite webhook

* Add BaseEvent, ShareEvent

* fix: Add share events to form

* fix: Move webhook delivery cleanup to single DB call
Remove some unused abstraction

* Add user, collection, group context to membership webhook events
Some associated refactoring

Co-authored-by: Corey Alexander <coreyja@gmail.com>
2022-06-28 22:44:50 -07:00
Tom Moor
68ce304b48 fix: Language in document notification email, missing collection name 2022-06-24 10:01:54 +02:00
Tom Moor
728790e38f feat: Validate Google, Azure, OIDC SSO access (#3590)
* chore: Store expiresAt on UserAuthentications. This represents the time that the accessToken is no longer valid and should be exchanged using the refreshToken

* feat: Check and expire Google SSO

* fix: Better handling of multiple auth methods
Added more docs

* fix: Retry access validation with network errors

* Small refactor, add Azure token validation support

* doc

* test

* lint

* OIDC refresh support

* CheckSSOAccessTask -> ValidateSSOAccessTask
Added lastValidatedAt column
Skip checks if validated within 5min
Some edge cases around encrypted columns
2022-06-05 13:18:51 -07:00
Tom Moor
bf0ff6c823 chore: Casing of logger -> Logger as it's an instantiated class 2022-05-21 13:59:23 +01:00
Tom Moor
3c002f82cc chore: Centralize env parsing, validation, defaults, and deprecation notices (#3487)
* chore: Centralize env parsing, defaults, deprecation

* wip

* test

* test

* tsc

* docs, more validation

* fix: Allow empty REDIS_URL (defaults to localhost)

* test

* fix: SLACK_MESSAGE_ACTIONS not bool

* fix: Add SMTP port validation
2022-05-19 08:05:11 -07:00
Tom Moor
bca66f7415 fix: Exports show as 0 bytes 2022-05-15 07:10:35 +01:00
Tom Moor
ec1bc801a4 fix: Write revision on document publish 2022-05-04 22:03:04 -07:00
Tom Moor
eeb8008927 chore: Refactor collection export to match import (#3483)
* chore: Refactor collection export to use FileOperations processor and task

* Tweak options
2022-05-01 21:06:07 -07:00
Tom Moor
c7e4f491eb fix: Correctly escape RegExp in import tasks
closes #3460
2022-04-27 23:58:01 -07:00
Tom Moor
576497eca1 feat: Allow override default Task options on schedule 2022-04-27 18:54:54 -07:00
Tom Moor
7436d4c5c1 fix: Truncate stored error message for file ops
closes #3459
2022-04-25 21:00:31 -07:00
Tom Moor
33ce49cc33 chore: Refactor data import (#3434)
* Complete refactor of import

* feat: Notion data import (#3442)
2022-04-23 10:07:35 -07:00
Tom Moor
b1aba32b62 chore: Bringing across edits from enterprise codebase 2022-04-16 19:46:01 -07:00
Tom Moor
86f1645199 feat: Automatic invite reminder email (#3354)
* feat: Add user flags concept, for tracking bits on a user

* feat: Example flag usage for user invite resend abuse

* wip

* test

* fix: Set correct flag
2022-04-12 20:12:33 -07:00
Tom Moor
f399c9d38c chore: More tracing cleanup 2022-04-11 08:04:13 -07:00
Tom Moor
27597727ee chore: More tracing cleanup 2022-04-10 20:11:18 -07:00
Tom Moor
963475d2b0 fix: Queue retry behavior (#3359)
* fix: Queue retry behavior

* Add default options for task queue
2022-04-10 17:50:42 -07:00
Tom Moor
5c24f9e1d5 chore: Email + mailer refactor (#3342)
* Huge email refactor

* fix: One rename too many

* comments
2022-04-07 16:50:04 -07:00
Tom Moor
dbfdcd6d23 chore: Refactor worker, emails and data cleanup to task system (#3337)
* Refactor worker, all emails on task system

* fix

* lint

* fix: Remove a bunch of expect-error comments in related tests

* refactor: Move work from utils.gc into tasks

* test

* Add tracing to tasks and processors
fix: DebounceProcessor triggering on all events
Event.add -> Event.schedule
2022-04-06 16:48:28 -07:00
Tom Moor
8490cab9b1 fix: Export collections produces empty zip, regressed in #3064 2022-02-09 19:40:40 -08:00
Tom Moor
d643c9453e feat: Import improvements (#3064)
* feat: Split and simplify import/export pages in prep for more options

* minor fixes

* File operations for imports

* test

* icons
2022-02-06 22:29:24 -08:00
Tom Moor
c5a11fe17b chore: Enable eslint to enforce curly (#3060) 2022-02-05 10:15:40 -08:00
Tom Moor
0bee0bbade fix: Remap event name for Slack integration 2022-02-02 21:31:51 -08:00
Tom Moor
79e2cad5b9 feat: Add reordering to starred documents (#2953)
* draft

* reordering

* JIT Index stars on first load

* test

* Remove unused code on client

* small unrefactor
2022-01-21 18:11:50 -08:00
Tom Moor
c6cc04cad8 chore: More typescript FIXME's removed 2022-01-06 21:25:42 -08:00
Tom Moor
b20a341f0c chore: Typescript database models (#2886)
closes #2798
2022-01-06 18:24:28 -08:00
Tom Moor
eb0c324da8 feat: Pin to home (#2880) 2021-12-30 16:54:02 -08:00
Tom Moor
d4695f3b5b feat: Add event selection to Slack post integration (#2857) 2021-12-16 22:30:23 -08:00
Tom Moor
32126602fd fix: Duplicate notification for publish event, closes #2757 2021-12-14 17:21:11 -08:00
Tom Moor
7c37724f88 types 2021-12-03 10:34:41 -06:00
Tom Moor
15b1069bcc chore: Move to Typescript (#2783)
This PR moves the entire project to Typescript. Due to the ~1000 ignores this will lead to a messy codebase for a while, but the churn is worth it – all of those ignore comments are places that were never type-safe previously.

closes #1282
2021-11-29 06:40:55 -08:00
Tom Moor
e899616081 fix: Changing team settings should update in other tabs 2021-11-09 17:28:59 -08:00
Tom Moor
5f00e1394d fix: Move notifications to be revision driven (#2709) 2021-10-31 18:36:16 -07:00
Tom Moor
b9346fe6ea fix: Minor collab adjusts 2021-10-13 22:01:30 -07:00
Tom Moor
83a61b87ed feat: Normalized server logging (#2567)
* feat: Normalize logging

* Remove scattered console.error + Sentry.captureException

* Remove mention of debug

* cleanup dev output

* Edge cases, docs

* Refactor: Move logger, metrics, sentry under 'logging' folder.
Trying to reduce the amount of things under generic 'utils'

* cleanup, last few console calls
2021-09-14 18:04:35 -07:00
Tom Moor
801f6681ba Collaborative editing (#1660) 2021-09-10 22:46:57 -07:00