Commit Graph

140 Commits

Author SHA1 Message Date
Alex
c754c860fd
feat(server) user-defined storage structure (#1098)
[Breaking] newly uploaded file will conform to the default structure of `{uploadLocation}/{userId}/year/year-month-day/filename.ext`
2022-12-16 14:26:12 -06:00
Alex
2876c7ff97
Fix(server) Microservice didn't trigger to remove user (#1090) 2022-12-10 18:01:29 -06:00
Jason Rasmussen
242165485d
fix(server): unique email database constraint (#1082) 2022-12-09 21:16:25 -06:00
Jason Rasmussen
5e680551b9
feat(server,web): migrate oauth settings from env to system config (#1061) 2022-12-09 14:51:42 -06:00
Alex
5de8ea162d
feat(server) Tagging system (#1046) 2022-12-05 11:56:44 -06:00
Jason Rasmussen
99854e90be
feat(server): link via profile.sub (#1055) 2022-12-03 21:59:24 -06:00
Alex Tran
6ed072f67b
Added migration needed for OIDC 2022-11-18 23:22:27 -06:00
Alex
8bc64be77b
feat: support iOS LivePhoto backup (#950) 2022-11-18 23:12:54 -06:00
Jason Rasmussen
b5d75e2016
feat(server,web): system config for admin (#959)
* feat: add admin config module for user configured config, uses it for ffmpeg

* feat: add api endpoint to retrieve admin config settings and values

* feat: add settings panel to admin page on web (wip)

* feat: add api endpoint to update the admin config

* chore: re-generate openapi spec after rebase

* refactor: move from admin config to system config naming

* chore: move away from UseGuards to new @Authenticated decorator

* style: dark mode styling for lists and fix conflicting colors

* wip: 2 column design, no edit button

* refactor: system config

* chore: generate open api

* chore: rm broken test

* chore: cleanup types

* refactor: config module names

Co-authored-by: Zack Pollard <zackpollard@ymail.com>
Co-authored-by: Zack Pollard <zack.pollard@moonpig.com>
2022-11-14 22:39:32 -06:00
Jason Rasmussen
d3c35ec9c5
feat(server,web): OIDC Implementation (#884)
* chore: merge

* feat: nullable password

* feat: server debugger

* chore: regenerate api

* feat: auto-register flag

* refactor: oauth endpoints

* chore: regenerate api

* fix: default scope configuration

* refactor: pass in redirect uri from client

* chore: docs

* fix: bugs

* refactor: auth services and user repository

* fix: select password

* fix: tests

* fix: get signing algorithm from discovery document

* refactor: cookie constants

* feat: oauth logout

* test: auth services

* fix: query param check

* fix: regenerate open-api
2022-11-14 20:24:25 -06:00
Zeeshan Khan
fe4b307fe6
feat(server,web): Delete and restore user from the admin portal (#935)
* delete and restore user from admin UI

* addressed review comments and fix e2e test

* added cron job to delete user, and some formatting changes

* addressed review comments

* adding missing queue registration
2022-11-07 15:53:47 -06:00
Zeeshan Khan
a6eea4d096
feat(web) add asset count stats on admin page (#843) 2022-10-23 16:54:54 -05:00
bo0tzz
87ba99755b Remove unused variable 2022-10-13 22:17:31 +02:00
bo0tzz
c03f860f8e Log a warning if JWT_SECRET key does not have enough bits 2022-10-13 21:54:29 +02:00
Alex
7587f858ae
feat(server/web) Add manual job trigger mechanism to the web (#767) 2022-10-06 11:25:54 -05:00
Zack Pollard
c33775b944
feat(server): missing exif extract nightly task (#754)
* fix: nightly reverse geocoding task checking for mapbox

* refactor: remove file size from image processor and queue data

* feat: add missing exif nightly job

* Remove filesize requirement in assetUploadedProcessorName queue insertion

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2022-09-28 05:41:50 -05:00
Zack Pollard
f377b64065
feat(server) Remove mapbox and use local reverse geocoding (#738)
* feat: local reverse geocoding implementation, removes mapbox

* Disable non-null tslintrule

* Disable non-null tslintrule

* Remove tsignore

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2022-09-22 21:50:05 -05:00
Alex
e79e92c60f
Added Log level to background service (#685) 2022-09-13 12:09:57 -05:00
Alex
858ad43d3b
fix(server): harden inserting process, self-healing timestamp info on bad timestamp (#682)
* fix(server): harden inserting process, self-healing timestamp info
2022-09-12 23:35:44 -05:00
Alex
b081eda76f
fix(server): change the createdAt and modifiedAt to the correct type in database (#591)
* Added migration files

* Remove type casting in sql query
2022-09-05 20:51:01 -05:00
Thanh Pham
a467936e73
feat(server): de-duplication (#557)
* feat(server): remove un-used deviceAssetId cols.

* feat(server): return 409 if asset is duplicated

* feat(server): replace old unique constaint

* feat(server): strip deviceId in file path

* feat(server): skip duplicate asset

* chore(server): revert changes

* fix(server): asset test spec

* fix(server): checksum generation for uploaded assets

* fix(server): make sure generation queue run after migraion

* feat(server): remove temp file

* chore(server): remove dead code
2022-09-05 14:45:38 -05:00
Thanh Pham
c76f7804ab
feat(server): generate checksum for previous uploaded assets (#558)
* feat(server): generate checksum for previous uploaded assets

* fix(server): typo
2022-09-02 08:32:21 -05:00
Thanh Pham
b80dca74ef
feat(server): calculate sha1 checksum (#525)
* feat(server): override multer storage

* feat(server): calc sha1 of uploaded file

* feat(server): add checksum into asset

* chore(server): add package-lock for mkdirp package

* fix(server): free hash stream

* chore(server): rollback this changes, not refactor here

* refactor(server): re-arrange import statement

* fix(server): make sure hash done before callback

* refactor(server): replace varchar to char for checksum, reserve pixelChecksum for future

* refactor(server): remove pixelChecksum

* refactor(server): convert checksum from string to bytea

* feat(server): add index to checksum

* refactor(): rollback package.json changes

* feat(server): remove uploaded file when progress fail

* feat(server): calculate hash in sequence
2022-08-31 09:27:17 -05:00
Alex
87f7b0849a
Added migration down for change exif file type 2022-08-26 09:13:11 -07:00
Alex
4596a8ee01
Change fileSizeInByte to bigint from int to handle large size (#534) 2022-08-26 09:07:59 -07:00
Thanh Pham
3b55cdc0be
refactor(server): move constant into common package (#522)
* refactor(server): move constant into common package

* refactor(server): re-arrange import statement in microservice module

* refactor(server): move app.config into common package

* fix(server): e2e testing
2022-08-23 07:34:21 -07:00
Thanh Pham
7f9f825589
fix(server): correct media info (#508)
* fix(server): correct media info

* fix(server): video metadata
2022-08-20 22:58:47 -07:00
Thanh Pham
9c30d58b10
feat(server): preserve caption fields and extract mediainfo for video (#505)
* feat(server): preserve caption fields and extract mediainfo for video

* Fixed Geocoding missing info leads to fail EXIF extraction for the whole file

Co-authored-by: Alex <alex.tran1502@gmail.com>
2022-08-20 16:31:37 -07:00
Alex Tran
e4c4b53fcd
Added imageName as searchable text on database 2022-07-26 13:43:12 -05:00
Alex Tran
d8b26c6da8
Update bug report template 2022-07-16 10:54:00 -05:00
Fynn Petersen-Frey
d69470e207
Add extended redis & DB port configuration via environment variables (#330)
* Add database port configuration via env variable

Add redis connection configuration via env variables

* Add redis connection configuration via env variables
2022-07-12 22:21:11 -05:00
Alex
357f7d1c31
Added schedule job to perform reverse geocoding if key is added after backing up assets (#305) 2022-07-04 15:16:39 -05:00
Zack Pollard
e6d30d72fa
Fix typeorm migrations (#297)
* fix: remove config parameter from typeorm cli and update config

the config parameter is no longer supported since version 0.3
the config now needs to export a DataSource object to work with the 0.3 cli

* fix: update all typeorm entities and migrations to be aligned with database structure

* Fixed test-util import databaseConfig

* Fixed column mismatch in raw query with new migration

* Remove dist build directory when starting dev server

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2022-07-04 14:20:43 -05:00
Alex
97d9b80baa
Added creation date for video from ffmpeg.prob (#303) 2022-07-04 13:44:43 -05:00
Alex
76bf1c0379
Remove thumbnail generation on mobile app (#292)
* Remove thumbnail generation on mobile

* Remove tconditions for missing thumbnail on the backend

* Remove console.log

* Refactor queue systems

* Convert queue and processor name to constant

* Added corresponding interface to job queue
2022-07-02 21:06:36 -05:00
Alex
a2e1d4caa2
Update server dependencies and fixed Typeorm API changes in new version (#276)
* Fixed dependencies

* Adapt typeorm API to be compatible with new version

* Fixed typeorm API in tests

* Remove console.log
2022-06-29 13:39:58 -05:00
Alex
5f00d8b9c6
Added mechanism of required password change of new user's first login (#272)
* Deprecate login scenarios that support pre-web era

* refactor and simplify setup

* Added user info to change password form

* change isFistLogin column to shouldChangePassword

* Implemented change user password

* Implement the change password page for mobile

* Change label

* Added changes log and up minor version

* Fixed typo in the release note

* Up server version
2022-06-27 15:13:07 -05:00
Jaime Baez
c918f5b001
Set TypeScript to strict mode and fix issues related to server types (#261)
* Fix lint issues and some other TS issues

- set TypeScript in strict mode
- add npm commands to lint / check code
- fix all lint issues
- fix some TS issues
- rename User reponse DTO to make it consistent with the other ones
- override Express/User interface to use UserResponseDto interface
 This is for when the accessing the `user` from a Express Request,
 like in `asset-upload-config`

* Fix the rest of TS issues

- fix all the remaining TypeScript errors
- add missing `@types/mapbox__mapbox-sdk` package

* Move global.d.ts to server `src` folder

* Update AssetReponseDto duration type

This is now of type `string` that defaults to '0:00:00.00000' if not set
which is what the mobile app currently expects

* Set context when logging error in asset.service

Use `ServeFile` as the context for logging an error when
asset.resizePath is not set

* Fix wrong AppController merge conflict resolution

`redirectToWebpage` was removed in main as is no longer used.
2022-06-25 12:53:06 -05:00
Jaime Baez
517a3363d6
Refactor API for albums feature (#155)
* Rename "shared" to "album"

Prepare moving "SharedAlbums" to "Albums"

* Update server album API endpoints

* Update mobile app album endpoints

Also add `putRequest` to mobile network.service

* Add GET album collection filter

- allow to filter by owner = 'mine' | 'their'
- make sharedWithUserIds no longer required when creating an album

* Rename remaining variables to "album"

* Add ParseMeUUIDPipe to validate uuid or `me`

* Add album params validation

* Update todo in mobile album service.

* Setup e2e testing

* Add user e2e tests

* Rename database host env variable to DB_HOST

* Add some `Album` e2e tests

Also fix issues found with the tests

* Force push (try to recover DB_HOST env)

* Rename db host env variable to `DB_HOSTNAME`

* Remove unnecessary `initDb` from test-utils

The current database.config is running the migrations:
`migrationsRun: true`

* Remove `initDb` usage from album e2e test

* Update GET albums filter to `shared`

- add filter by all / shared / not shared
- add response DTOs
- add GET albums e2e tests

* Update album e2e tests for user.service changes

* Update mobile app to use album response DTOs

* Refactor album-service DB into album-registry

- DB logic refactored into album-repository making it easier to test
- add some album-service unit tests
- add `clearMocks` to jest configuration

* Finish implementing album.service unit tests

* Rename response DTO

Make them consistent with rest of the project naming

* Update debug log messages in mobile network service

* Rename table `shared_albums` to `albums`

* Rename table `asset_shared_album`

* Rename Albums `sharedAssets` to `assets`

* Update tests to match updated "delete" response

* Fixed asset cannot be compared in Set by adding Equatable package

* Remove hero effect to fixed janky animation

Co-authored-by: Alex <alex.tran1502@gmail.com>
2022-06-18 10:56:36 -05:00
Alex
a8220172f8
WIP refactor container and queuing system (#206)
* refactor microservices to machine-learning

* Update tGithub issue template with correct task syntax

* Added microservices container

* Communicate between service based on queue system

* added dependency

* Fixed problem with having to import BullQueue into the individual service

* Added todo

* refactor server into monorepo with microservices

* refactor database and entity to library

* added simple migration

* Move migrations and database config to library

* Migration works in library

* Cosmetic change in logging message

* added user dto

* Fixed issue with testing not able to find the shared library

* Clean up library mapping path

* Added webp generator to microservices

* Update Github Action build latest

* Fixed issue NPM cannot install due to conflict witl Bull Queue

* format project with prettier

* Modified docker-compose file

* Add GH Action for Staging build:

* Fixed GH action job name

* Modified GH Action to only build & push latest when pushing to main

* Added Test 2e2 Github Action

* Added Test 2e2 Github Action

* Implemented microservice to extract exif

* Added cronjob to scan and generate webp thumbnail  at midnight

* Refactor to ireduce hit time to database when running microservices

* Added error handling to asset services that handle read file from disk

* Added video transcoding queue to process one video at a time

* Fixed loading spinner on web while loading covering the info panel

* Add mechanism to show new release announcement to web and mobile app (#209)

* Added changelog page

* Fixed issues based on PR comments

* Fixed issue with video transcoding run on the server

* Change entry point content for backward combatibility when starting up server

* Added announcement box

* Added error handling to failed silently when the app version checking is not able to make the request to GITHUB

* Added new version announcement overlay

* Update message

* Added messages

* Added logic to check and show announcement

* Add method to handle saving new version

* Added button to dimiss the acknowledge message

* Up version for deployment to the app store
2022-06-11 16:12:06 -05:00