From 2593bbdb73c87fdc09ac0d21b37dabe459e4ea3f Mon Sep 17 00:00:00 2001 From: funkecoder23 <12570656+FunkeCoder23@users.noreply.github.com> Date: Sun, 4 Feb 2024 15:11:39 -0500 Subject: [PATCH 1/5] update readme with more info add info about installing docker update DMM setup with correct path for env reorder env setup and DMM setup add info about pgloader and combine SQL command command into a docker exec use github markdown for notes and important info --- README.md | 79 ++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 52 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index 5e649e2..decda5f 100644 --- a/README.md +++ b/README.md @@ -5,15 +5,18 @@ A self-hosted Stremio addon for streaming torrents via a debrid service. ## Contents -**Note: Until we reach `v1.0.0`, please consider releases as alpha.** +> [!CAUTION] +> Until we reach `v1.0.0`, please consider releases as alpha. -**Important: The latest change renames the project and requires a [small migration](#selfhostio-to-knightcrawler-migration).** +> [!IMPORTANT] +> The latest change renames the project and requires a [small migration](#selfhostio-to-knightcrawler-migration). - [Knight Crawler](#knight-crawler) - [Contents](#contents) - [Overview](#overview) - [Using](#using) - - [Initial setup (optional)](#initial-setup-optional) + - [Download Docker and Docker Compose v2](#download-docker-and-docker-compose-v2) - [Environment Setup](#environment-setup) + - [DebridMediaManager setup (optional)](#debridmediamanager-setup-optional) - [Run the project](#run-the-project) - [Monitoring with Grafana and Prometheus (Optional)](#monitoring-with-grafana-and-prometheus-optional) - [Accessing RabbitMQ Management](#accessing-rabbitmq-management) @@ -36,9 +39,24 @@ Stremio is a media player. On it's own it will not allow you to watch anything. The project is shipped as an all-in-one solution. The initial configuration is designed for hosting only on your local network. If you want it to be accessible from outside of your local network, please see [not yet available]() -### Initial setup (optional) +### Download Docker and Docker Compose v2 -After cloning the repository there are some steps you should take to maximise the number of movies/tv shows we can find. +Download and install [Docker Compose](https://docs.docker.com/compose/install/), bundled with [Docker Desktop](https://docs.docker.com/desktop/) or, if using Linux, you can install [Docker Engine](https://docs.docker.com/engine/install/) and the [Docker Compose Plugin.](https://docs.docker.com/compose/install/linux/) + +### Environment Setup + +Before running the project, you need to set up the environment variables. Copy the `.env.example` file to `.env`: + +```sh +cd deployment/docker +cp .env.example .env +``` + +Then set any of the values you wouldd like to customize. + +### DebridMediaManager setup (optional) + +There are some optional steps you should take to maximise the number of movies/tv shows we can find. We can search DebridMediaManager hash lists which are hosted on GitHub. This allows us to add hundreds of thousands of movies and tv shows, but it requires a Personal Access Token to be generated. The software only needs read access and only for public respositories. To generate one, please follow these steps: @@ -56,26 +74,15 @@ We can search DebridMediaManager hash lists which are hosted on GitHub. This all (checked) Public Repositories (read-only) ``` 4. Click `Generate token` -5. Take the new token and add it to the bottom of the [env/producer.env](env/producer.env) file +5. Take the new token and add it to the bottom of the [.env](.env) file ``` GithubSettings__PAT= ``` -### Environment Setup - -Before running the project, you need to set up the environment variables. Copy the `.env.example` file to `.env`: - -```sh -cd deployment/docker -cp .env.example .env -``` - -Then set any of th values you'd like to customize. - ### Run the project -Open a terminal in the directory and run the command: +Open a terminal in the project directory and run the command: ```sh cd deployment/docker @@ -121,7 +128,8 @@ Here's how to set up and use Grafana and Prometheus for monitoring RabbitMQ: Now, you can use these dashboards to monitor RabbitMQ and Postgres metrics. -Note: If you encounter issues with missing or unavailable data in Grafana, please ensure on [Prometheus's target page](http://127.0.0.1:9090/targets) that the RabbitMQ target is up and running. +> [!NOTE] +> If you encounter issues with missing or unavailable data in Grafana, please ensure on [Prometheus's target page](http://127.0.0.1:9090/targets) that the RabbitMQ target is up and running. ## Importing external dumps @@ -130,7 +138,10 @@ A brief record of the steps required to import external data, in this case the r ### Import data into database -I created a file `db.load` as follows.. + +Using [pgloader](https://pgloader.readthedocs.io/en/latest/ref/sqlite.html) we can import other databases into Knight Crawler. + +For example, create a file called `db.load` containing the following: ``` load database @@ -142,16 +153,27 @@ with include drop, create tables, create indexes, reset sequences set work_mem to '16MB', maintenance_work_mem to '512 MB'; ``` -And ran `pgloader db.load` to create a new `items` table. +Then run `pgloader db.load` to create a new `items` table. ### INSERT INTO ingested_torrents -Once the `items` table is available in the postgres database, put all the tv/movie items into the `ingested_torrents` table using: + +> [!NOTE] +> This is specific to this example external database, other databases may/will have different column names and the sql command will require tweaking + +> [!IMPORTANT] +> The `processed` field should be false so that the consumers will properly process it. + + +Once the `items` table is available in the postgres database, put all the tv/movie items into the `ingested_torrents` table using `psql`. + +This can be done by attaching to the postgres docker container ``` -INSERT INTO ingested_torrents (name, source, category, info_hash, size, seeders, leechers, imdb, processed, "createdAt", "updatedAt") -SELECT title, 'RARBG', cat, hash, size, NULL, NULL, imdb, false, current_timestamp, current_timestamp -FROM items where cat='tv' OR cat='movies'; +docker exec -it knightcrawler-postgres-1 psql -d knightcrawler -c " +INSERT INTO ingested_torrents (name, source, category, info_hash, size, seeders, leechers, imdb, processed) +SELECT title, 'RARBG', cat, hash, size, NULL, NULL, imdb, false +FROM items where cat='tv' OR cat='movies';" ``` ## Selfhostio to KnightCrawler Migration @@ -160,11 +182,14 @@ With the renaming of the project, you will have to change your database name in **With your existing stack still running**, run: ``` -docker exec -it torrentio-selfhostio-postgres-1 psql -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid <> pg_backend_pid() AND datname = 'selfhostio'; ALTER DATABASE selfhostio RENAME TO knightcrawler;" +docker exec -it torrentio-selfhostio-postgres-1 psql -c " +SELECT pg_terminate_backend(pid) FROM pg_stat_activity +WHERE pid <> pg_backend_pid() AND datname = 'selfhostio'; +ALTER DATABASE selfhostio RENAME TO knightcrawler;" ``` Make sure your postgres container is named `torrentio-selfhostio-postgres-1`, otherwise, adjust accordingly. -This command should return: `ALTER DATABASE`. This means your database is now renamed. You can now pull the new changes if you haven't already and run `docker-compose up -d`. +This command should return: `ALTER DATABASE`. This means your database is now renamed. You can now pull the new changes if you haven't already and run `docker compose up -d`. ## To-do From 101b5c416e35a7e196e5fd5d2e350e7bb22b7205 Mon Sep 17 00:00:00 2001 From: funkecoder23 <12570656+FunkeCoder23@users.noreply.github.com> Date: Sun, 4 Feb 2024 16:25:38 -0500 Subject: [PATCH 2/5] Add optional configuration section Add details about consumer replicas Add CONSUMER_REPLICAS to .env.example Change `consumer: deploy: replicas` to use CONSUMER_REPLICAS var to keep user config options in one file --- README.md | 11 +++++++++++ deployment/docker/.env.example | 1 + deployment/docker/docker-compose.yaml | 2 +- 3 files changed, 13 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index decda5f..f4f8fb9 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,7 @@ A self-hosted Stremio addon for streaming torrents via a debrid service. - [Using](#using) - [Download Docker and Docker Compose v2](#download-docker-and-docker-compose-v2) - [Environment Setup](#environment-setup) + - [Optional Configuration Changes](#optional-configuration-changes) - [DebridMediaManager setup (optional)](#debridmediamanager-setup-optional) - [Run the project](#run-the-project) - [Monitoring with Grafana and Prometheus (Optional)](#monitoring-with-grafana-and-prometheus-optional) @@ -54,6 +55,16 @@ cp .env.example .env Then set any of the values you wouldd like to customize. +#### Optional Configuration Changes + +> [!WARNING] +> These values should be tested and tuned for your specific machine. + +By default, Knight Crawler is configured to be *relatively* conservative in its resource usage. If running on a decent machine (16GB RAM, i5+ or equivalent), you can increase some settings to increase consumer throughput. This is especially helpful if you have a large backlog from [importing databases](#importing-external-dumps). + +In your `.env` file, under the `# Consumer` section increase `CONSUMER_REPLICAS` from `3` to `15`. +You can also increase `JOB_CONCURRENCY` from `5` to `10`. + ### DebridMediaManager setup (optional) There are some optional steps you should take to maximise the number of movies/tv shows we can find. diff --git a/deployment/docker/.env.example b/deployment/docker/.env.example index 21a715c..ee06905 100644 --- a/deployment/docker/.env.example +++ b/deployment/docker/.env.example @@ -26,6 +26,7 @@ JOBS_ENABLED=true MAX_SINGLE_TORRENT_CONNECTIONS=10 TORRENT_TIMEOUT=30000 UDP_TRACKERS_ENABLED=true +CONSUMER_REPLICAS=3 # Producer RabbitMqConfiguration__Host=rabbitmq diff --git a/deployment/docker/docker-compose.yaml b/deployment/docker/docker-compose.yaml index f03fb37..86fbb0e 100644 --- a/deployment/docker/docker-compose.yaml +++ b/deployment/docker/docker-compose.yaml @@ -101,7 +101,7 @@ services: labels: logging: "promtail" deploy: - replicas: 3 + replicas: ${CONSUMER_REPLICAS} <<: *knightcrawler-app networks: - knightcrawler-network From 2d2792681643df334efed4be7ce54f80ced3fbba Mon Sep 17 00:00:00 2001 From: funkecoder23 <12570656+FunkeCoder23@users.noreply.github.com> Date: Sun, 4 Feb 2024 19:44:18 -0500 Subject: [PATCH 3/5] correct the psql command escaped quotes to properly parse the "createdAt" and "updatedAt" column names --- README.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index f4f8fb9..74d413b 100644 --- a/README.md +++ b/README.md @@ -110,6 +110,7 @@ To enhance your monitoring capabilities, you can use Grafana and Prometheus in a #### Accessing RabbitMQ Management + You can still monitor RabbitMQ by accessing its management interface at [http://127.0.0.1:15672/](http://127.0.0.1:15672/). Use the provided credentials to log in and explore RabbitMQ's monitoring features (the default username and password are `guest`). #### Using Grafana and Prometheus @@ -152,18 +153,22 @@ A brief record of the steps required to import external data, in this case the r Using [pgloader](https://pgloader.readthedocs.io/en/latest/ref/sqlite.html) we can import other databases into Knight Crawler. -For example, create a file called `db.load` containing the following: +For example, if you had a sql database called `rarbg_db.sqlite` stored in `/tmp/` you would create a file called `db.load` containing the following: ``` load database - from sqlite:///tmp/rarbg_db.sqlite - into postgresql://postgres:postgres@localhost/knightcrawler + from sqlite://tmp/rarbg_db.sqlite + into postgresql://postgres:postgres@/knightcrawler with include drop, create tables, create indexes, reset sequences set work_mem to '16MB', maintenance_work_mem to '512 MB'; ``` +> [!TIP] +> Your `docker-ip` can be found using the following command: +> `docker network inspect knightcrawler-network | grep knightcrawler-postgres -A 4` + Then run `pgloader db.load` to create a new `items` table. ### INSERT INTO ingested_torrents @@ -173,17 +178,16 @@ Then run `pgloader db.load` to create a new `items` table. > This is specific to this example external database, other databases may/will have different column names and the sql command will require tweaking > [!IMPORTANT] -> The `processed` field should be false so that the consumers will properly process it. - +> The `processed` field should be `false` so that the consumers will properly process it. Once the `items` table is available in the postgres database, put all the tv/movie items into the `ingested_torrents` table using `psql`. -This can be done by attaching to the postgres docker container +This can be done by running the following command: ``` docker exec -it knightcrawler-postgres-1 psql -d knightcrawler -c " -INSERT INTO ingested_torrents (name, source, category, info_hash, size, seeders, leechers, imdb, processed) -SELECT title, 'RARBG', cat, hash, size, NULL, NULL, imdb, false +INSERT INTO ingested_torrents (name, source, category, info_hash, size, seeders, leechers, imdb, processed, \"createdAt\", \"updatedAt\") +SELECT title, 'RARBG', cat, hash, size, NULL, NULL, imdb, false, current_timestamp, current_timestamp FROM items where cat='tv' OR cat='movies';" ``` From 46da28f10c79c4f7a59f8ee322cdaaae13d00fa0 Mon Sep 17 00:00:00 2001 From: funkecoder23 <12570656+FunkeCoder23@users.noreply.github.com> Date: Sun, 4 Feb 2024 20:30:31 -0500 Subject: [PATCH 4/5] change .env path to match new location --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 74d413b..1853665 100644 --- a/README.md +++ b/README.md @@ -85,7 +85,7 @@ We can search DebridMediaManager hash lists which are hosted on GitHub. This all (checked) Public Repositories (read-only) ``` 4. Click `Generate token` -5. Take the new token and add it to the bottom of the [.env](.env) file +5. Take the new token and add it to the bottom of the [.env](deployment/docker/.env) file ``` GithubSettings__PAT= ``` @@ -157,7 +157,7 @@ For example, if you had a sql database called `rarbg_db.sqlite` stored in `/tmp/ ``` load database - from sqlite://tmp/rarbg_db.sqlite + from sqlite:///tmp/rarbg_db.sqlite into postgresql://postgres:postgres@/knightcrawler with include drop, create tables, create indexes, reset sequences From 73cf8351058e3a26ee1780e02aa264835df7698b Mon Sep 17 00:00:00 2001 From: funkecoder23 <12570656+FunkeCoder23@users.noreply.github.com> Date: Sun, 4 Feb 2024 21:03:15 -0500 Subject: [PATCH 5/5] add information on dropping items table after import --- README.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1853665..9f5e2f1 100644 --- a/README.md +++ b/README.md @@ -167,9 +167,11 @@ with include drop, create tables, create indexes, reset sequences > [!TIP] > Your `docker-ip` can be found using the following command: -> `docker network inspect knightcrawler-network | grep knightcrawler-postgres -A 4` +> ``` +> docker network inspect knightcrawler-network | grep knightcrawler-postgres -A 4 +> ``` -Then run `pgloader db.load` to create a new `items` table. +Then run `pgloader db.load` to create a new `items` table. This can take a few minutes, depending on the size of the database. ### INSERT INTO ingested_torrents @@ -191,6 +193,15 @@ SELECT title, 'RARBG', cat, hash, size, NULL, NULL, imdb, false, current_timesta FROM items where cat='tv' OR cat='movies';" ``` +You should get a response similar to: + +`INSERT 0 669475` + +After, you can delete the `items` table by running: + +```docker exec -it knightcrawler-postgres-1 psql -d knightcrawler -c "drop table items";``` + + ## Selfhostio to KnightCrawler Migration With the renaming of the project, you will have to change your database name in order to keep your existing data.