diff --git a/README.md b/README.md index 79d9e95..e4ceb5a 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,9 @@ Join our [Discord](https://discord.gg/8fQdxay9z2)! - [Using Grafana and Prometheus](#using-grafana-and-prometheus) - [Importing external dumps](#importing-external-dumps) - [Import data into database](#import-data-into-database) + - [Alternative: Using Docker](#alternative-using-docker) - [INSERT INTO ingested\_torrents](#insert-into-ingested_torrents) + - [Fixing imported databases](#fixing-imported-databases) - [Selfhostio to KnightCrawler Migration](#selfhostio-to-knightcrawler-migration) - [To-do](#to-do) @@ -175,7 +177,7 @@ with include drop, create tables, create indexes, reset sequences Then run `pgloader db.load` to create a new `items` table. This can take a few minutes, depending on the size of the database. -### Import data into database (ALTERNATIVE: Using Docker) +#### Alternative: Using Docker Move your sql database called `rarbg_db.sqlite` and `db.load` into your current working directoy @@ -200,22 +202,30 @@ docker run --rm -it --network=knightcrawler-network -v "$(pwd)":/data dimitri/pg ### INSERT INTO ingested_torrents - > [!NOTE] > This is specific to this example external database, other databases may/will have different column names and the sql command will require tweaking > [!IMPORTANT] > The `processed` field should be `false` so that the consumers will properly process it. -Once the `items` table is available in the postgres database, put all the tv/movie items into the `ingested_torrents` table using `psql`. +Once the `items` table is available in the postgres database, put all the tv/movie items into the `ingested_torrents` table using `psql`. +Because this specific database also contains categories such as `tv_uhd` we can use a `LIKE` query and coerce it into the `tv` category. This can be done by running the following command: ``` docker exec -it knightcrawler-postgres-1 psql -d knightcrawler -c " -INSERT INTO ingested_torrents (name, source, category, info_hash, size, seeders, leechers, imdb, processed, "createdAt", "updatedAt") -SELECT title, 'RARBG', cat, hash, size, NULL, NULL, imdb, false, current_timestamp, current_timestamp -FROM items WHERE NOT EXISTS (SELECT info_hash FROM ingested_torrents WHERE ingested_torrents.info_hash = items.hash) AND (cat LIKE 'tv%' OR cat LIKE 'movies%');" +INSERT INTO ingested_torrents (name, source, category, info_hash, size, seeders, leechers, imdb, processed, \"createdAt\", \"updatedAt\") +SELECT title, 'RARBG', 'tv', hash, size, NULL, NULL, imdb, false, current_timestamp, current_timestamp +FROM items where cat LIKE 'tv%%' ON CONFLICT DO NOTHING;" +``` + +And similarly for movies: +``` +docker exec -it knightcrawler-postgres-1 psql -d knightcrawler -c " +INSERT INTO ingested_torrents (name, source, category, info_hash, size, seeders, leechers, imdb, processed, \"createdAt\", \"updatedAt\") +SELECT title, 'RARBG', 'movies', hash, size, NULL, NULL, imdb, false, current_timestamp, current_timestamp +FROM items where cat LIKE 'movies%%' ON CONFLICT DO NOTHING;" ``` You should get a response similar to: @@ -227,6 +237,14 @@ After, you can delete the `items` table by running: ```docker exec -it knightcrawler-postgres-1 psql -d knightcrawler -c "drop table items";``` +#### Fixing imported databases + +If you've already imported a database with incorrect categories, you can fix these categories by running the following commands: + +```docker exec -it knightcrawler-postgres-1 psql -d knightcrawler -c "UPDATE ingested_torrents SET category='movies', processed='f' WHERE category LIKE 'movies_%';"``` + +```docker exec -it knightcrawler-postgres-1 psql -d knightcrawler -c "UPDATE ingested_torrents SET category='movies', processed='f' WHERE category LIKE 'movies_%';"``` + ## Selfhostio to KnightCrawler Migration With the renaming of the project, you will have to change your database name in order to keep your existing data.