docs: Update documentation (#6430)

* Documentation corrections

* fix import

* add firewall note

* npm run format:fix

* fixs

* npm run format:fix

* space

* fix note

* admin-jobs.png image update + fixes

* Storage Template.md update

* Add new Troubleshooting about symbolic link in library

* Updating the libraries.md

* Updating the libraries.md

* Corrections

* add `/`

* ...

* Add Python script to remove-offline-files.md

* npm run format:fix

* Add info about HDR in FAQ

* My wrong merge

* add info about symlink

* [Community] + PowerShell

* add 360 photo support to Features in README

* add info about remote ML and info about orphaned files from the external library to the scripts page

* Typo

* add note about storage locations

* add info about Purge for portainer and link to info about asset types and storage locations

* npm run format:fix

* Add FAQ about "faces" that aren't faces

* Update docs/docs/administration/backup-and-restore.md

* Update docs/docs/administration/backup-and-restore.md

---------

Co-authored-by: Jason Rasmussen <jrasm91@gmail.com>

README.md

@@ -71,10 +71,12 @@ Spec: Free-tier Oracle VM - Amsterdam - 2.4Ghz quad-core ARM64 CPU, 24GB RAM
 
 ## Features
 
+
 | Features                                     | Mobile | Web |
-| -------------------------------------------- | ------ | --- |
+| :--------------------------------------------- | -------- | ----- |
 | Upload and view videos and photos            | Yes    | Yes |
 | Auto backup when the app is opened           | Yes    | N/A |
+| Prevent duplication of assets                | Yes    | Yes |
 | Selective album(s) for backup                | Yes    | N/A |
 | Download photos and videos to local device   | Yes    | Yes |
 | Multi-user support                           | Yes    | Yes |
@@ -89,6 +91,7 @@ Spec: Free-tier Oracle VM - Amsterdam - 2.4Ghz quad-core ARM64 CPU, 24GB RAM
 | OAuth support                                | Yes    | Yes |
 | API Keys                                     | N/A    | Yes |
 | LivePhoto/MotionPhoto backup and playback    | Yes    | Yes |
+| Support 360 degree image display             | No     | Yes |
 | User-defined storage structure               | Yes    | Yes |
 | Public Sharing                               | No     | Yes |
 | Archive and Favorites                        | Yes    | Yes |
@@ -118,6 +121,7 @@ If you feel like this is the right cause and the app is something you are seeing
 - ZCash: u1smm4wvqegcp46zss2jf5xptchgeczp4rx7a0wu3mermf2wxahm26yyz5w9mw3f2p4emwlljxjumg774kgs8rntt9yags0whnzane4n67z4c7gppq4yyvcj404ne3r769prwzd9j8ntvqp44fa6d67sf7rmcfjmds3gmeceff4u8e92rh38nd30cr96xw6vfhk6scu4ws90ldzupr3sz
 
 ## Contributors
+
 <a href="https://github.com/alextran1502/immich/graphs/contributors">
   <img src="https://contrib.rocks/image?repo=immich-app/immich" width="100%"/>
 </a>

docs/docs/FAQ.mdx

@@ -86,6 +86,10 @@ Also, there are additional jobs for person (face) thumbnails.
 
 There are no requirements for assets to be unique across users. If multiple users upload the same image they are processed as if they were distinct assets and jobs run and thumbnails are generated accordingly.
 
+### Why do HDR videos appear pale in Immich player but look normal after download?
+
+Immich uses a player with known HDR color display issues. We are experimenting with a different player that provides better color profiles for HDR content for future improvements.
+
 ### How can I delete transcoded videos without deleting the original?
 
 The transcode of an asset can be deleted by setting a transcode policy that makes it unnecessary, then running a transcoding job for that asset. This can be done on a per-asset basis by starting a transcoding job for an asset (with the _Refresh encoded videos_ button in the asset viewer options. Or, for all assets by running transcoding jobs for all assets.
@@ -210,6 +214,11 @@ On the other hand, Immich does scan video thumbnails for faces, so it can perfor
 
 No.
 
+### I'm getting a lot of "faces" that aren't faces, what can I do?
+
+You can increase the MIN DETECTION SCORE to 0.8 to help prevent bad thumbnails. However, a score of 0.9 might filter out too many real faces depending on the library used. If you just want to hide specific faces, you can adjust the 'MIN FACES DETECTED' setting in the administration panel  
+to increase the bar for what the algorithm considers a "core face" for that person, reducing the chance of bad thumbnails being chosen.
+
 ### The immich_model-cache volume takes up a lot of space, what could be the problem?
 
 If you installed several models and chose not to use some of them, it might be worth deleting the old models that are in immich_model-cache.
@@ -286,7 +295,7 @@ The non-root user/group needs read/write access to the volume mounts, including
 Data for Immich comes in two forms:
 
 1. **Metadata** stored in a postgres database, persisted via the `pg_data` volume
-2. **Files** (originals, thumbs, profile, etc.), stored in the `UPLOAD_LOCATION` folder.
+2. **Files** (originals, thumbs, profile, etc.), stored in the `UPLOAD_LOCATION` folder, more [info](/docs/administration/backup-and-restore#asset-types-and-storage-locations).
 
 To remove the **Metadata** you can stop Immich and delete the volume.
 
@@ -294,6 +303,11 @@ To remove the **Metadata** you can stop Immich and delete the volume.
 docker compose down -v
 ```
 
+:::note Portainer
+If you use portainer, bring down the stack in portainer. Go into the volumes section  
+and remove all the volumes related to immcih then restart the stack.
+:::
+
 After removing the containers and volumes, the **Files** can be cleaned up (if necessary) from the `UPLOAD_LOCATION` by simply deleting any unwanted files or folders.
 
 ### Why does the machine learning service report workers crashing?

docs/docs/administration/backup-and-restore.md

@@ -1,5 +1,8 @@
 # Backup and Restore
 
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
 A [3-2-1 backup strategy](https://www.backblaze.com/blog/the-3-2-1-backup-strategy/) is recommended to protect your data. You should keep copies of your uploaded photos/videos as well as the Immich database for a comprehensive backup solution. This page provides an overview on how to backup the database and the location of user-uploaded pictures and videos. A template bash script that can be run as a cron job is provided [here](/docs/guides/template-backup-script.md)
 
 ## Database
@@ -14,7 +17,10 @@ Refer to the official [postgres documentation](https://www.postgresql.org/docs/c
 
 The recommended way to backup and restore the Immich database is to use the `pg_dumpall` command.
 
-```bash title='Backup'
+<Tabs>
+  <TabItem value="Linux system based Backup" label="Linux system based Backup" default>
+
+```bash title='Bash'
 docker exec -t immich_postgres pg_dumpall -c -U postgres | gzip > "/path/to/backup/dump.sql.gz"
 ```
 
@@ -28,6 +34,26 @@ gunzip < "/path/to/backup/dump.sql.gz" | docker exec -i immich_postgres psql -U
 docker compose up -d    # Start remainder of Immich apps
 ```
 
+</TabItem>
+  <TabItem value="Windows system based Backup" label="Windows system based Backup">
+
+```powershell title='Backup'
+docker exec -t immich_postgres pg_dumpall -c -U postgres > "\path\to\backup\dump.sql"
+```
+
+```powershell title='Restore'
+docker compose down -v  # CAUTION! Deletes all Immich data to start from scratch.
+docker compose pull     # Update to latest version of Immich (if desired)
+docker compose create   # Create Docker containers for Immich apps without running them.
+docker start immich_postgres    # Start Postgres server
+sleep 10    # Wait for Postgres server to start up
+gc "C:\path\to\backup\dump.sql" | docker exec -i immich_postgres psql -U postgres -d immich    # Restore Backup
+docker compose up -d    # Start remainder of Immich apps
+```
+
+</TabItem>
+</Tabs>
+
 Note that for the database restore to proceed properly, it requires a completely fresh install (i.e. the Immich server has never run since creating the Docker containers). If the Immich app has run, Postgres conflicts may be encountered upon database restoration (relation already exists, violated foreign key constraints, multiple primary keys, etc.).
 
 The database dumps can also be automated (using [this image](https://github.com/prodrigestivill/docker-postgres-backup-local)) by editing the docker compose file to match the following:
@@ -64,34 +90,78 @@ gunzip < db_dumps/last/immich-latest.sql.gz | docker exec -i immich_postgres psq
 Immich stores two types of content in the filesystem: (1) original, unmodified content, and (2) generated content. Only the original content needs to be backed-up, which includes the following folders:
 
 1. `UPLOAD_LOCATION/library`
-1. `UPLOAD_LOCATION/upload`
+2. `UPLOAD_LOCATION/upload`
-1. `UPLOAD_LOCATION/profile`
+3. `UPLOAD_LOCATION/profile`
+
+### Asset Types and Storage Locations
+
+Some storage locations are impacted by the Storage Template. See below for more details.
+
+<Tabs>
+  <TabItem value="Storage Template Off (Default)." label="Storage Template Off (Default)." default>
+
+:::note
+`UPLOAD_LOCATION/library` folder is not used by default on new machines running version 1.92.0. These are if the system administrator activated the storage template engine, for [more info](https://github.com/immich-app/immich/releases#:~:text=the%20partner%E2%80%99s%20assets.-,Hardening%20storage%20template,-We%20have%20further).
+:::
 
 **1. User-Specific Folders:**
 
 - Each user has a unique string representing them.
-  - The main user is "Admin" (but only for `\library\library\`)
+- You can find your user ID in Account Account Settings -> Account -> User ID.
-  - Other users have different string identifiers.
-- You can find your user ID in Account Account Settings > Account > User ID.
 
 **2. Asset Types and Storage Locations:**
 
 - **Source Assets:**
-  - Original assets uploaded through the browser interface&mobile&CLI.
+  - Original assets uploaded through the browser interface & mobile & CLI.
-  - Stored in `\library\library\<userID>`.
+  - Stored in `/library/upload/<userID>`.
 - **Avatar Images:**
   - User profile images.
-  - Stored in `\library\profile\<userID>`.
+  - Stored in `/library/profile/<userID>`.
 - **Thumbs Images:**
   - Preview images (blurred, small, large) for each asset and thumbnails for recognized faces.
-  - Stored in `\library\thumbs\<userID>`.
+  - Stored in `/library/thumbs/<userID>`.
+- **Encoded Assets:**
+  - By default, unless otherwise specified re-encoded video assets for wider compatibility.
+  - Stored in `/library/encoded-video/<userID>`.
+
+</TabItem>
+  <TabItem value="Storage Template On" label="Storage Template On">
+
+:::note
+If you choose to activate the storage template engine, it will move all assets to `UPLOAD_LOCATION/library/<userID>`.
+
+When you turn off the storage template engine, it will leave the assets in `UPLOAD_LOCATION/library/<userID>` and will not return them to `/library/upload`.  
+**New assets** will be saved to `/library/upload`.
+:::
+
+**1. User-Specific Folders:**
+
+- Each user has a unique string representing them.
+  - The main user is "Admin" (but only for `UPLOAD_LOCATION/library`)
+  - Other users have different string identifiers.
+- You can find your user ID in Account Account Settings -> Account -> User ID.
+
+**2. Asset Types and Storage Locations:**
+
+- **Source Assets:**
+  - Original assets uploaded through the browser interface & mobile & CLI.
+  - Stored in `UPLOAD_LOCATION/library/<userID>`.
+- **Avatar Images:**
+  - User profile images.
+  - Stored in `/library/profile/<userID>`.
+- **Thumbs Images:**
+  - Preview images (blurred, small, large) for each asset and thumbnails for recognized faces.
+  - Stored in `/library/thumbs/<userID>`.
 - **Encoded Assets:**
   - By default, unless otherwise specified re-encoded video assets for wider compatibility .
-  - Stored in `\library\encoded-video\<userID>`.
+  - Stored in `/library/encoded-video/<userID>`.
 - **Files in Upload Queue (Mobile):**
   - Files uploaded through mobile apps.
-  - Temporarily located in `\library\upload\<userID>`.
+  - Temporarily located in `/library/upload/<userID>`.
-  - Transferred to `\library\library\<userID>` upon successful upload.
+  - Transferred to `UPLOAD_LOCATION/library/<userID>` upon successful upload.
+
+</TabItem>
+</Tabs>
 
 :::danger
 Do not touch the files inside these folders under any circumstances except taking a backup, changing or removing an asset can cause untracked and missing files.

docs/docs/features/libraries.md

@@ -52,8 +52,11 @@ Sometimes, an external library will not scan correctly. This can happen if the i
 - In the docker-compose file, are the volumes mounted correctly?
 - Are the volumes identical between the `server` and `microservices` container?
 - Are the import paths set correctly, and do they match the path set in docker-compose file?
+- Are you using symbolic link in your import library?
 - Are the permissions set correctly?
 - Are you using forward slashes everywhere? (`/`)
+- Are you using symlink across docker mounts?
+- Are you using [spaces in the internal path](/docs/features/libraries#:~:text=can%20be%20accessed.-,NOTE,-Spaces%20in%20the)?
 
 If all else fails, you can always start a shell inside the container and check if the path is accessible. For example, `docker exec -it immich_microservices /bin/bash` will start a bash shell. If your import path, for instance, is `/data/import/photos`, you can check if the files are accessible by running `ls /data/import/photos`. Also check the `immich_server` container in the same way.
 
@@ -121,7 +124,9 @@ First, we need to plan how we want to organize the libraries. The christmas trip
 The `ro` flag at the end only gives read-only access to the volumes. While Immich does not modify files, it's a good practice to mount read-only.
 :::
 
-_Remember to bring the container down/up to register the changes. Make sure you can see the mounted path in the container._
+:::info
+_Remember to bring the container `docker compose down/up` to register the changes. Make sure you can see the mounted path in the container._
+:::
 
 ### Set External Path
 

docs/docs/guides/database-gui.md

@@ -4,8 +4,17 @@ A short guide on connecting [pgAdmin](https://www.pgadmin.org/) to Immich.
 
 :::note
 
-- In order to connect to the database the immich_postgres container **must be running**.
+In order to connect to the database the immich_postgres container **must be running**.
-- The passwords and usernames used below match the ones specified in the example `.env` file. If changed, please use actual values instead.
+
+The passwords and usernames used below match the ones specified in the example `.env` file. If changed, please use actual values instead.
+
+**Optional:** To connect to the database **outside** of your Docker's network:
+
+- Expose port 5432 in your `docker-compose.yml` file.
+- Edit the PostgreSQL [`pg_hba.conf`](https://www.postgresql.org/docs/current/auth-pg-hba-conf.html) file.
+- Make sure your firewall does not block access to port 5432.
+  Note that exposing the database port increases the risk of getting attacked by hackers.  
+  Make sure to remove the binding port after finishing the database's tasks.
 
 :::
 

docs/docs/guides/remote-machine-learning.md

@@ -6,6 +6,10 @@ To alleviate [performance issues on low-memory systems](/docs/FAQ.mdx#why-is-imm
 - Copy the following `docker-compose.yml` to your ML system.
 - Start the container by running `docker-compose up -d` or `docker compose up -d` (depending on your Docker version).
 
+:::note Info
+Starting with version v1.93.0 face detection work and face recognize were split. From now on face detection is done in the immich_machine_learning service, but facial recognition is done in the immich_microservices service.
+:::
+
 ```yaml
 version: '3.8'
 

docs/docs/guides/remove-offline-files.md

@@ -1,11 +1,146 @@
-# Remove Offline Files
+# Remove Offline Files [Community]
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
 
 :::note
-**Before running the script**, please make sure you have a [backup](/docs/administration/backup-and-restore) of your assets and database
+**Before running the script**, please make sure you have a [backup](/docs/administration/backup-and-restore) of your assets and database.
+:::
+
+:::info
+**None** of the scripts can delete orphaned files from the external library.
 :::
 
 This page is a guide to get rid of offline files from the repair page.
 
+<Tabs>
+
+<TabItem value="Python script (Best way)" label="Python script (Best way)">
+
+This way works by retrieving a file that contains a list of all the files that are defined as offline files, running a script that uses the [Immich API](/docs/api/delete-assets) in order to remove the offline files.
+
+1. Create an API key under Admin User -> Account Settings -> API Keys -> New API Key -> Copy to clipboard.
+2. Copy and save the code to file -> `Immich Remove Offline Files.py`.
+3. Run the script and follow the instructions.
+
+:::note
+You might need to run `pip install halo tabulate tqdm` if these dependencies are missing on your machine.
+:::
+
+```bash title='Python'
+#!/usr/bin/env python3
+
+# Note: you might need to run "pip install halo tabulate tqdm" if these dependencies are missing on your machine
+
+import argparse
+import json
+import requests
+
+from datetime import datetime
+from halo import Halo
+from tabulate import tabulate
+from tqdm import tqdm
+from urllib.parse import urlparse
+
+def parse_arguments():
+    parser = argparse.ArgumentParser(description='Fetch file report and delete orphaned media assets from Immich.')
+    parser.add_argument('--apikey', help='Immich API key for authentication')
+    parser.add_argument('--immichaddress', help='Full address for Immich, including protocol and port')
+    parser.add_argument('--no_prompt', action='store_true', help='Delete orphaned media assets without confirmation')
+    args = parser.parse_args()
+    return args
+
+def filter_entities(response_json, entity_type):
+    return [
+        {'pathValue': entity['pathValue'], 'entityId': entity['entityId'], 'entityType': entity['entityType']}
+        for entity in response_json.get('orphans', []) if entity.get('entityType') == entity_type
+    ]
+
+def main():
+    args = parse_arguments()
+    try:
+        if args.apikey:
+            api_key = args.apikey
+        else:
+            api_key = input('Enter the Immich API key: ')
+
+        if args.immichaddress:
+            immich_server = args.immichaddress
+        else:
+            immich_server = input('Enter the full web address for Immich, including protocol and port: ')
+        immich_parsed_url = urlparse(immich_server)
+        base_url = f'{immich_parsed_url.scheme}://{immich_parsed_url.netloc}'
+        api_url = f'{base_url}/api'
+        file_report_url = api_url + '/audit/file-report'
+        headers = {'x-api-key': api_key}
+
+        print()
+        spinner = Halo(text='Retrieving list of orphaned media assets...', spinner='dots')
+        spinner.start()
+
+        try:
+            response = requests.get(file_report_url, headers=headers)
+            response.raise_for_status()
+            spinner.succeed('Success!')
+        except requests.exceptions.RequestException as e:
+            spinner.fail(f'Failed to fetch assets: {str(e)}')
+
+        person_assets = filter_entities(response.json(), 'person')
+        orphan_media_assets = filter_entities(response.json(), 'asset')
+
+        num_entries = len(orphan_media_assets)
+
+        if num_entries == 0:
+            print('No orphaned media assets found; exiting.')
+            return
+
+        else:
+            if not args.no_prompt:
+                table_data = []
+                for asset in orphan_media_assets:
+                    table_data.append([asset['pathValue'], asset['entityId']])
+                print(tabulate(table_data, headers=['Path Value', 'Entity ID'], tablefmt='pretty'))
+                print()
+
+                if person_assets:
+                    print('Found orphaned person assets! Please run the "RECOGNIZE FACES > ALL" job in Immich after running this tool to correct this.')
+                    print()
+
+                if num_entries > 0:
+                    summary = f'There {"is" if num_entries == 1 else "are"} {num_entries} orphaned media asset{"s" if num_entries != 1 else ""}. Would you like to delete {"them" if num_entries != 1 else "it"} from Immich? (yes/no): '
+                    user_input = input(summary).lower()
+                    print()
+
+                    if user_input not in ('y', 'yes'):
+                        print('Exiting without making any changes.')
+                        return
+
+            with tqdm(total=num_entries, desc="Deleting orphaned media assets", unit="asset") as progress_bar:
+                for asset in orphan_media_assets:
+                    entity_id = asset['entityId']
+                    asset_url = f'{api_url}/asset'
+                    delete_payload = json.dumps({'force': True, 'ids': [entity_id]})
+                    headers = {'Content-Type': 'application/json', 'x-api-key': api_key}
+                    response = requests.delete(asset_url, headers=headers, data=delete_payload)
+                    response.raise_for_status()
+                    progress_bar.set_postfix_str(entity_id)
+                    progress_bar.update(1)
+            print()
+            print('Orphaned media assets deleted successfully!')
+    except Exception as e:
+        print()
+        print(f"An error occurred: {str(e)}")
+
+if __name__ == '__main__':
+    main()
+```
+
+Thanks to [DooMRunneR](https://discord.com/channels/979116623879368755/1179655214870040596/1194308198413373482) and [Sircharlo](https://discord.com/channels/979116623879368755/1179655214870040596/1195038609812758639) for writing this script.
+
+</TabItem>
+
+<TabItem value="Bash and PowerShell script" label="Bash and PowerShell script" default>
+
 This way works by downloading a JSON file that contains a list of all the files that are defined as offline files, running a script that uses the [Immich API](/docs/api/delete-assets) in order to remove the offline files.
 
 1. Create an API key under Admin User -> Account Settings -> API Keys -> New API Key -> Copy to clipboard.
@@ -15,13 +150,13 @@ This way works by downloading a JSON file that contains a list of all the files
 
 ## Script for Linux based systems:
 
-```bash
+```bash title='Bash'
 awk -F\" '/entityId/ {print $4}' orphans.json | while read line; do curl --location --request DELETE 'http://YOUR_IP_HERE:2283/api/asset' --header 'Content- Type: application/json' --header 'x-api-key: YOUR_API_KEY_HERE' --data '{ "force": true, "ids": ["'"$line"'"]}';done
 ```
 
 ## Script for the Windows system (run through PowerShell):
 
-```powershell
+```powershell title='PowerShell'
 Get-Content orphans.json | Select-String -Pattern 'entityId' | ForEach-Object {
   $line = $_ -split '"' | Select-Object -Index 3
   $body = [pscustomobject]@{
@@ -36,3 +171,6 @@ Get-Content orphans.json | Select-String -Pattern 'entityId' | ForEach-Object {
 ```
 
 Thanks to [DooMRunneR](https://discord.com/channels/979116623879368755/1179655214870040596/1194308198413373482) for writing this script.
+
+</TabItem>
+</Tabs>

docs/docs/partials/_storage-template.md

@@ -1,5 +1,12 @@
 Immich allows the admin user to set the uploaded filename pattern. Both at the directory and filename level.
 
+:::note new version
+On new machines running version 1.92.0 storage template engine is off by default, for [more info](https://github.com/immich-app/immich/releases#:~:text=the%20partner%E2%80%99s%20assets.-,Hardening%20storage%20template,-We%20have%20further).
+:::
+
+:::tip
+You can read more about the differences between storage template engine on and off [here](/docs/administration/backup-and-restore#asset-types-and-storage-locations)
+:::
 The admin user can set the template by using the template builder in the `Administration -> Settings -> Storage Template`. Immich provides a set of variables that you can use in constructing the template, along with additional custom text. If the template produces [multiple files with the same filename, they won't be overwritten](https://github.com/immich-app/immich/discussions/3324) as a sequence number is appended to the filename.
 
 ```bash title="Default template"