Commit Graph

168 Commits

Author SHA1 Message Date
Jason Rasmussen
8f304b8157
refactor(server): shared links (#1385)
* refactor(server): shared links

* chore: tests

* fix: bugs and tests

* fix: missed one expired at

* fix: standardize file upload checks

* test: lower flutter version

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2023-01-25 10:35:28 -06:00
Skyler Mäntysaari
9d337bf4dc
feat(server/machine-learning): Configurable port (#1386)
* feat(server/machine-learning): Configurable port

* feat(server/machine-learning): Address PR comments.

* feat(server/machine-learning): Simplify

Co-authored-by: Alex <alex.tran1502@gmail.com>
2023-01-23 22:18:35 -06:00
Jason Rasmussen
eade36ee82
refactor(server): auth service (#1383)
* refactor: auth

* chore: tests

* Remove await on non-async method

* refactor: constants

* chore: remove extra async

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2023-01-23 22:13:42 -06:00
Alex
b07891089f
feat(web/server) Add more options to public shared link (#1348)
* Added migration files

* Added logic for shared album level

* Added permission for EXIF

* Update shared link response dto

* Added condition to show download button

* Create and edit link with new parameter:

* Remove deadcode

* PR feedback

* More refactor

* Move logic of allow original file to service

* Simplify

* Wording
2023-01-21 22:15:16 -06:00
Jason Rasmussen
4cfac47674
refactor(server): job repository (#1382)
* refactor(server): job repository

* refactor: job repository

* chore: generate open-api

* fix: job panel

* Remove incorrect subtitle

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2023-01-21 22:13:36 -06:00
Zack Pollard
4e0fe27de3
feat(server): transcoding improvements (#1370)
* feat: support isEdited flag for SettingSwitch

* feat: add transcodeAll ffmpeg settings for extra transcoding control

* refactor: tidy up and rename current video transcoding code + transcode everything

* feat: better video transcoding with ffprobe

analyses video files to see if they are already in the desired format
allows admin to choose to transcode all videos regardless of the current format

* fix: always serve encoded video if it exists

* feat: change video codec option to a select box, limit options

removed previous video codec config option as it's incompatible with new options
removed mapping for encoder to codec as we now store the codec in the config

* feat: add video conversion job for transcoding previously missed videos

* chore: fix spelling of job messages to pluralise assets

* chore: fix prettier/eslint warnings

* feat: force switch targetAudioCodec default to aac to avoid iOS incompatibility

* chore: lint issues after rebase
2023-01-21 20:09:02 -06:00
Skyler Mäntysaari
8eb82836b9
feat(server): Support webm videos (#1365)
* feat(server): Support webm without transcoding.

Transcoding result doesn't appear to be used by anything expect for quicktime.

* feat(server): Fix the asset uploader for .avi

It needs to be transcoded.

* feat(server): Most browsers doesn't support avi so use mp4.

* feat(server): Address PR comments

* Addressed the PR comments

I moved the function that checks the mimetype to a central location in asset-utils and made tests for it.

* Rollbacked to the way transcoder was decising things to transcode.
2023-01-21 15:52:40 -06:00
Jason Rasmussen
c0a6b3d5a3
refactor(server): system config (#1353)
* refactor(server): system config

* fix: jest circular import

* chore: ignore migrations in coverage report

* chore: tests

* chore: tests

* chore: todo note

* chore: remove vite config backup

* chore: fix redis hostname
2023-01-21 10:11:55 -06:00
Hammer
5340683199
Allow the use of SSL connections to the postgres database. (#1256)
* Allow the use of SSL connections to the postgres database.

* Add default SSL false when no env set

* Add commented out example of DB_SSL env

* Refactor add SSL option into PostgresConnectionOptions

* Refactor the database connection to optionally use a URL string instead of the env variables

* Refactor the database connection based on feedback

* Add dynamic validation around the DB envs

* Remove DB_URL from example

* Fix rebase

* Add back the optional database port in the example

* Formatted file correctly

* change types to a const to fix tests
2023-01-20 14:27:01 -06:00
Jason Rasmussen
bdad18a572
feat(server): turn off machine learning endpoint (#1361) 2023-01-20 10:35:55 -06:00
Jason Rasmussen
92972ac776
refactor(server): api keys (#1339)
* refactor: api keys

* refactor: test module

* chore: tests

* chore: fix provider

* refactor: test mock repos
2023-01-18 08:40:15 -06:00
Alex
3e4a14b299
chore(server) harden EXIF extraction (#1347)
* chore(server) Harden EXIF extraction

* Remove unused function in timeutil

* Remove deadcode
2023-01-17 13:41:00 -06:00
Skyler Mäntysaari
dff10e89fe
feat(server): Fix exif data parsing (#1326)
* Trying to get exifdata working with different lib.

* Got the new library working.

* Addressing PR comments.

* Removed not used vars and proper place for the eslint disable.

* Fix time-utils to use the exiftool-vendored lib.

Fixed also one test, as that would be valid.

* Using filename for timestamp as well if possible.

* Add new tests for time-utils.

* Remember to gracefully terminate the exiftool instance when not needed.

* eslint ignore...

* Apperantly Dockerfile changes were not pushed.

* feat(dockerfile): Tweak the Server Dockerfile

* feat(server): getTimestampFromFilename should return string or undefined.

* feat(server): If we don't have exifData or timestamp from filename, raise an error.

* Apparently test was already right, but my local system disagrees.

* More utilities for parsing and fix the timestampFromFilename.

It was returning an incorrect date as the regex doesn't seem to be the best for this as files named `IMG_0115.HEIC` will want to get parsed incorrectly due to it.

* feat(server/docker): Install perl as it seems to be required.

* feat(server): remember to include exposureTime and focalLength in new exif data.

* feat(server): Remove the parsing from filename as requested.

* feat(server): Import exiftool differently in time-utils.

* feat(server): Error handling when there is no exifData.

* feat(server): Fixes for the error handling when there is no exifData.

* feat(server): Remember to include modifyDate despite no exif.

* feat(server): Remember to include model of Camera.

* feat(server): Fixing up Exiftool usage.

Including proper logging for it, which had to be done in wrapped fashion due to it expecting all the logging levels which NextJS logger doesn't implement.

* feat(server): Do not use a wrapper for ExifTool logging.

* fix merge conflicts in metadata-extractor
2023-01-17 09:29:49 -06:00
Jason Rasmussen
693adf8488
refactor: job names (#1343)
* refactor: job names

* refactor: remove jobId
2023-01-17 08:43:45 -06:00
Alex
0c582df962
feat(server) Add filetype variable to storage template (#1337)
* feat(server) Add filetype variable to storage template

* Remove console.log

* Added additional variable for full file type
2023-01-16 15:54:52 -06:00
Jason Rasmussen
1e2f02613f
refactor: reset admin password (#1335)
* refactor: reset-admin-password

* chore: docs
2023-01-16 12:09:04 -06:00
bo0tzz
b9b2b559a1
fix(database): Set connection timeout (#1324) 2023-01-14 09:06:59 -06:00
Jason Rasmussen
ba04b753de
refactor: logging (#1318) 2023-01-13 08:23:12 -06:00
Jason Rasmussen
755a1331da
chore(web,server): run code coverage reports (#1313)
* chore(web,server): run code coverage reports

* chore(tests): fail test check if coverage drops

* chore: disable e2e until they are fixed

* chore(web): coverage threshold
2023-01-12 16:07:57 -06:00
Jason Rasmussen
131caa20eb
refactor(server): domain/infra (#1298)
* refactor: user repository

* refactor: user module

* refactor: move database into infra

* refactor(cli): use user core

* chore: import path

* chore: tests
2023-01-11 21:34:36 -05:00
Jason Rasmussen
bd838a71d1
feat(web,server): disable password login (#1223)
* feat(web,server): disable password login

* chore: unit tests

* chore: fix import

* chore: linting

* feat(cli): server command for enable/disable password login

* chore: update docs

* feat(web): confirm dialogue

* chore: linting

* chore: linting

* chore: linting

* chore: linting

* chore: linting

* chore: fix web test

* chore: server unit tests
2023-01-09 16:32:58 -05:00
Alex
10789503c1
feat(web/server) public album sharing (#1266) 2023-01-09 14:16:08 -06:00
Jason Rasmussen
9e6d6b2532
feat(web,server): api keys (#1244)
* feat(server): api keys

* chore: open-api

* feat(web): api keys

* fix: remove keys when deleting a user
2023-01-02 14:22:33 -06:00
Jason Rasmussen
d9fa6619e7
refactor(server): imports and repository tokens (#1220)
* refactor: entity imports

* refactor: rename user repository token

* chore: merge imports

* refactor: rename album repository token

* refactor: rename asset repository token

* refactor: rename tag repository token
2022-12-30 07:22:06 -06:00
Jason Rasmussen
6974d4068b
feat(server): mobile oauth with custom scheme redirect uri (#1204)
* feat(server): support providers without support for custom schemas

* chore: unit tests

* chore: test mobile override

* chore: add details to the docs
2022-12-29 14:47:30 -06:00
Jason Rasmussen
4e860b024b
refactor(server): drop salt column (#1185) 2022-12-26 23:03:14 -05:00
Jason Rasmussen
7dc12dea1e
feat(web,server): link/unlink oauth account (#1154)
* feat(web,server): link/unlink oauth account

* chore: linting

* fix: broken oauth callback

* fix: user core bugs

* fix: tests

* fix: use user response

* chore: update docs

* feat: prevent the same oauth account from being linked twice

* chore: mock logger
2022-12-26 09:35:52 -06:00
Alex
de69d0031e
chore(server) Add job for storage migration (#1117) 2022-12-19 12:13:10 -06:00
Alex
c754c860fd
feat(server) user-defined storage structure (#1098)
[Breaking] newly uploaded file will conform to the default structure of `{uploadLocation}/{userId}/year/year-month-day/filename.ext`
2022-12-16 14:26:12 -06:00
Alex
2876c7ff97
Fix(server) Microservice didn't trigger to remove user (#1090) 2022-12-10 18:01:29 -06:00
Jason Rasmussen
242165485d
fix(server): unique email database constraint (#1082) 2022-12-09 21:16:25 -06:00
Jason Rasmussen
5e680551b9
feat(server,web): migrate oauth settings from env to system config (#1061) 2022-12-09 14:51:42 -06:00
Alex
5de8ea162d
feat(server) Tagging system (#1046) 2022-12-05 11:56:44 -06:00
Jason Rasmussen
99854e90be
feat(server): link via profile.sub (#1055) 2022-12-03 21:59:24 -06:00
Alex Tran
6ed072f67b
Added migration needed for OIDC 2022-11-18 23:22:27 -06:00
Alex
8bc64be77b
feat: support iOS LivePhoto backup (#950) 2022-11-18 23:12:54 -06:00
Jason Rasmussen
b5d75e2016
feat(server,web): system config for admin (#959)
* feat: add admin config module for user configured config, uses it for ffmpeg

* feat: add api endpoint to retrieve admin config settings and values

* feat: add settings panel to admin page on web (wip)

* feat: add api endpoint to update the admin config

* chore: re-generate openapi spec after rebase

* refactor: move from admin config to system config naming

* chore: move away from UseGuards to new @Authenticated decorator

* style: dark mode styling for lists and fix conflicting colors

* wip: 2 column design, no edit button

* refactor: system config

* chore: generate open api

* chore: rm broken test

* chore: cleanup types

* refactor: config module names

Co-authored-by: Zack Pollard <zackpollard@ymail.com>
Co-authored-by: Zack Pollard <zack.pollard@moonpig.com>
2022-11-14 22:39:32 -06:00
Jason Rasmussen
d3c35ec9c5
feat(server,web): OIDC Implementation (#884)
* chore: merge

* feat: nullable password

* feat: server debugger

* chore: regenerate api

* feat: auto-register flag

* refactor: oauth endpoints

* chore: regenerate api

* fix: default scope configuration

* refactor: pass in redirect uri from client

* chore: docs

* fix: bugs

* refactor: auth services and user repository

* fix: select password

* fix: tests

* fix: get signing algorithm from discovery document

* refactor: cookie constants

* feat: oauth logout

* test: auth services

* fix: query param check

* fix: regenerate open-api
2022-11-14 20:24:25 -06:00
Zeeshan Khan
fe4b307fe6
feat(server,web): Delete and restore user from the admin portal (#935)
* delete and restore user from admin UI

* addressed review comments and fix e2e test

* added cron job to delete user, and some formatting changes

* addressed review comments

* adding missing queue registration
2022-11-07 15:53:47 -06:00
Zeeshan Khan
a6eea4d096
feat(web) add asset count stats on admin page (#843) 2022-10-23 16:54:54 -05:00
bo0tzz
87ba99755b Remove unused variable 2022-10-13 22:17:31 +02:00
bo0tzz
c03f860f8e Log a warning if JWT_SECRET key does not have enough bits 2022-10-13 21:54:29 +02:00
Alex
7587f858ae
feat(server/web) Add manual job trigger mechanism to the web (#767) 2022-10-06 11:25:54 -05:00
Zack Pollard
c33775b944
feat(server): missing exif extract nightly task (#754)
* fix: nightly reverse geocoding task checking for mapbox

* refactor: remove file size from image processor and queue data

* feat: add missing exif nightly job

* Remove filesize requirement in assetUploadedProcessorName queue insertion

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2022-09-28 05:41:50 -05:00
Zack Pollard
f377b64065
feat(server) Remove mapbox and use local reverse geocoding (#738)
* feat: local reverse geocoding implementation, removes mapbox

* Disable non-null tslintrule

* Disable non-null tslintrule

* Remove tsignore

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2022-09-22 21:50:05 -05:00
Alex
e79e92c60f
Added Log level to background service (#685) 2022-09-13 12:09:57 -05:00
Alex
858ad43d3b
fix(server): harden inserting process, self-healing timestamp info on bad timestamp (#682)
* fix(server): harden inserting process, self-healing timestamp info
2022-09-12 23:35:44 -05:00
Alex
b081eda76f
fix(server): change the createdAt and modifiedAt to the correct type in database (#591)
* Added migration files

* Remove type casting in sql query
2022-09-05 20:51:01 -05:00
Thanh Pham
a467936e73
feat(server): de-duplication (#557)
* feat(server): remove un-used deviceAssetId cols.

* feat(server): return 409 if asset is duplicated

* feat(server): replace old unique constaint

* feat(server): strip deviceId in file path

* feat(server): skip duplicate asset

* chore(server): revert changes

* fix(server): asset test spec

* fix(server): checksum generation for uploaded assets

* fix(server): make sure generation queue run after migraion

* feat(server): remove temp file

* chore(server): remove dead code
2022-09-05 14:45:38 -05:00
Thanh Pham
c76f7804ab
feat(server): generate checksum for previous uploaded assets (#558)
* feat(server): generate checksum for previous uploaded assets

* fix(server): typo
2022-09-02 08:32:21 -05:00
Thanh Pham
b80dca74ef
feat(server): calculate sha1 checksum (#525)
* feat(server): override multer storage

* feat(server): calc sha1 of uploaded file

* feat(server): add checksum into asset

* chore(server): add package-lock for mkdirp package

* fix(server): free hash stream

* chore(server): rollback this changes, not refactor here

* refactor(server): re-arrange import statement

* fix(server): make sure hash done before callback

* refactor(server): replace varchar to char for checksum, reserve pixelChecksum for future

* refactor(server): remove pixelChecksum

* refactor(server): convert checksum from string to bytea

* feat(server): add index to checksum

* refactor(): rollback package.json changes

* feat(server): remove uploaded file when progress fail

* feat(server): calculate hash in sequence
2022-08-31 09:27:17 -05:00
Alex
87f7b0849a
Added migration down for change exif file type 2022-08-26 09:13:11 -07:00
Alex
4596a8ee01
Change fileSizeInByte to bigint from int to handle large size (#534) 2022-08-26 09:07:59 -07:00
Thanh Pham
3b55cdc0be
refactor(server): move constant into common package (#522)
* refactor(server): move constant into common package

* refactor(server): re-arrange import statement in microservice module

* refactor(server): move app.config into common package

* fix(server): e2e testing
2022-08-23 07:34:21 -07:00
Thanh Pham
7f9f825589
fix(server): correct media info (#508)
* fix(server): correct media info

* fix(server): video metadata
2022-08-20 22:58:47 -07:00
Thanh Pham
9c30d58b10
feat(server): preserve caption fields and extract mediainfo for video (#505)
* feat(server): preserve caption fields and extract mediainfo for video

* Fixed Geocoding missing info leads to fail EXIF extraction for the whole file

Co-authored-by: Alex <alex.tran1502@gmail.com>
2022-08-20 16:31:37 -07:00
Alex Tran
e4c4b53fcd
Added imageName as searchable text on database 2022-07-26 13:43:12 -05:00
Alex Tran
d8b26c6da8
Update bug report template 2022-07-16 10:54:00 -05:00
Fynn Petersen-Frey
d69470e207
Add extended redis & DB port configuration via environment variables (#330)
* Add database port configuration via env variable

Add redis connection configuration via env variables

* Add redis connection configuration via env variables
2022-07-12 22:21:11 -05:00
Alex
357f7d1c31
Added schedule job to perform reverse geocoding if key is added after backing up assets (#305) 2022-07-04 15:16:39 -05:00
Zack Pollard
e6d30d72fa
Fix typeorm migrations (#297)
* fix: remove config parameter from typeorm cli and update config

the config parameter is no longer supported since version 0.3
the config now needs to export a DataSource object to work with the 0.3 cli

* fix: update all typeorm entities and migrations to be aligned with database structure

* Fixed test-util import databaseConfig

* Fixed column mismatch in raw query with new migration

* Remove dist build directory when starting dev server

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2022-07-04 14:20:43 -05:00
Alex
97d9b80baa
Added creation date for video from ffmpeg.prob (#303) 2022-07-04 13:44:43 -05:00
Alex
76bf1c0379
Remove thumbnail generation on mobile app (#292)
* Remove thumbnail generation on mobile

* Remove tconditions for missing thumbnail on the backend

* Remove console.log

* Refactor queue systems

* Convert queue and processor name to constant

* Added corresponding interface to job queue
2022-07-02 21:06:36 -05:00
Alex
a2e1d4caa2
Update server dependencies and fixed Typeorm API changes in new version (#276)
* Fixed dependencies

* Adapt typeorm API to be compatible with new version

* Fixed typeorm API in tests

* Remove console.log
2022-06-29 13:39:58 -05:00
Alex
5f00d8b9c6
Added mechanism of required password change of new user's first login (#272)
* Deprecate login scenarios that support pre-web era

* refactor and simplify setup

* Added user info to change password form

* change isFistLogin column to shouldChangePassword

* Implemented change user password

* Implement the change password page for mobile

* Change label

* Added changes log and up minor version

* Fixed typo in the release note

* Up server version
2022-06-27 15:13:07 -05:00
Jaime Baez
c918f5b001
Set TypeScript to strict mode and fix issues related to server types (#261)
* Fix lint issues and some other TS issues

- set TypeScript in strict mode
- add npm commands to lint / check code
- fix all lint issues
- fix some TS issues
- rename User reponse DTO to make it consistent with the other ones
- override Express/User interface to use UserResponseDto interface
 This is for when the accessing the `user` from a Express Request,
 like in `asset-upload-config`

* Fix the rest of TS issues

- fix all the remaining TypeScript errors
- add missing `@types/mapbox__mapbox-sdk` package

* Move global.d.ts to server `src` folder

* Update AssetReponseDto duration type

This is now of type `string` that defaults to '0:00:00.00000' if not set
which is what the mobile app currently expects

* Set context when logging error in asset.service

Use `ServeFile` as the context for logging an error when
asset.resizePath is not set

* Fix wrong AppController merge conflict resolution

`redirectToWebpage` was removed in main as is no longer used.
2022-06-25 12:53:06 -05:00
Jaime Baez
517a3363d6
Refactor API for albums feature (#155)
* Rename "shared" to "album"

Prepare moving "SharedAlbums" to "Albums"

* Update server album API endpoints

* Update mobile app album endpoints

Also add `putRequest` to mobile network.service

* Add GET album collection filter

- allow to filter by owner = 'mine' | 'their'
- make sharedWithUserIds no longer required when creating an album

* Rename remaining variables to "album"

* Add ParseMeUUIDPipe to validate uuid or `me`

* Add album params validation

* Update todo in mobile album service.

* Setup e2e testing

* Add user e2e tests

* Rename database host env variable to DB_HOST

* Add some `Album` e2e tests

Also fix issues found with the tests

* Force push (try to recover DB_HOST env)

* Rename db host env variable to `DB_HOSTNAME`

* Remove unnecessary `initDb` from test-utils

The current database.config is running the migrations:
`migrationsRun: true`

* Remove `initDb` usage from album e2e test

* Update GET albums filter to `shared`

- add filter by all / shared / not shared
- add response DTOs
- add GET albums e2e tests

* Update album e2e tests for user.service changes

* Update mobile app to use album response DTOs

* Refactor album-service DB into album-registry

- DB logic refactored into album-repository making it easier to test
- add some album-service unit tests
- add `clearMocks` to jest configuration

* Finish implementing album.service unit tests

* Rename response DTO

Make them consistent with rest of the project naming

* Update debug log messages in mobile network service

* Rename table `shared_albums` to `albums`

* Rename table `asset_shared_album`

* Rename Albums `sharedAssets` to `assets`

* Update tests to match updated "delete" response

* Fixed asset cannot be compared in Set by adding Equatable package

* Remove hero effect to fixed janky animation

Co-authored-by: Alex <alex.tran1502@gmail.com>
2022-06-18 10:56:36 -05:00
Alex
a8220172f8
WIP refactor container and queuing system (#206)
* refactor microservices to machine-learning

* Update tGithub issue template with correct task syntax

* Added microservices container

* Communicate between service based on queue system

* added dependency

* Fixed problem with having to import BullQueue into the individual service

* Added todo

* refactor server into monorepo with microservices

* refactor database and entity to library

* added simple migration

* Move migrations and database config to library

* Migration works in library

* Cosmetic change in logging message

* added user dto

* Fixed issue with testing not able to find the shared library

* Clean up library mapping path

* Added webp generator to microservices

* Update Github Action build latest

* Fixed issue NPM cannot install due to conflict witl Bull Queue

* format project with prettier

* Modified docker-compose file

* Add GH Action for Staging build:

* Fixed GH action job name

* Modified GH Action to only build & push latest when pushing to main

* Added Test 2e2 Github Action

* Added Test 2e2 Github Action

* Implemented microservice to extract exif

* Added cronjob to scan and generate webp thumbnail  at midnight

* Refactor to ireduce hit time to database when running microservices

* Added error handling to asset services that handle read file from disk

* Added video transcoding queue to process one video at a time

* Fixed loading spinner on web while loading covering the info panel

* Add mechanism to show new release announcement to web and mobile app (#209)

* Added changelog page

* Fixed issues based on PR comments

* Fixed issue with video transcoding run on the server

* Change entry point content for backward combatibility when starting up server

* Added announcement box

* Added error handling to failed silently when the app version checking is not able to make the request to GITHUB

* Added new version announcement overlay

* Update message

* Added messages

* Added logic to check and show announcement

* Add method to handle saving new version

* Added button to dimiss the acknowledge message

* Up version for deployment to the app store
2022-06-11 16:12:06 -05:00