Merge pull request #81 from FunkeCoder23/insert-non-exact-categories

Add psql command for inserting non-exact movies and tv categories

README.md

@@ -26,7 +26,9 @@ Join our [Discord](https://discord.gg/8fQdxay9z2)!
       - [Using Grafana and Prometheus](#using-grafana-and-prometheus)
   - [Importing external dumps](#importing-external-dumps)
     - [Import data into database](#import-data-into-database)
+      - [Alternative: Using Docker](#alternative-using-docker)
     - [INSERT INTO ingested\_torrents](#insert-into-ingested_torrents)
+      - [Fixing imported databases](#fixing-imported-databases)
   - [Selfhostio to KnightCrawler Migration](#selfhostio-to-knightcrawler-migration)
   - [To-do](#to-do)
 
@@ -175,7 +177,7 @@ with include drop, create tables, create indexes, reset sequences
 
 Then run `pgloader db.load` to create a new `items` table. This can take a few minutes, depending on the size of the database.
 
-### Import data into database (ALTERNATIVE: Using Docker)
+#### Alternative: Using Docker
 
 Move your sql database called `rarbg_db.sqlite` and `db.load` into your current working directoy
 
@@ -200,22 +202,30 @@ docker run --rm -it --network=knightcrawler-network -v "$(pwd)":/data dimitri/pg
 
 ### INSERT INTO ingested_torrents
 
-
 > [!NOTE]
 > This is specific to this example external database, other databases may/will have different column names and the sql command will require tweaking
 
 > [!IMPORTANT]
 > The `processed` field should be `false` so that the consumers will properly process it.
 
-Once the `items` table is available in the postgres database, put all the tv/movie items into the `ingested_torrents` table using `psql`.
+Once the `items` table is available in the postgres database, put all the tv/movie items into the `ingested_torrents` table using `psql`. 
+Because this specific database also contains categories such as `tv_uhd` we can use a `LIKE` query and coerce it into the `tv` category.
 
 This can be done by running the following command:
 
 ```
 docker exec -it knightcrawler-postgres-1 psql -d knightcrawler -c "
-INSERT INTO ingested_torrents (name, source, category, info_hash, size, seeders, leechers, imdb, processed, "createdAt", "updatedAt")
+INSERT INTO ingested_torrents (name, source, category, info_hash, size, seeders, leechers, imdb, processed, \"createdAt\", \"updatedAt\")
-SELECT title, 'RARBG', cat, hash, size, NULL, NULL, imdb, false, current_timestamp, current_timestamp
+SELECT title, 'RARBG', 'tv', hash, size, NULL, NULL, imdb, false, current_timestamp, current_timestamp
-FROM items WHERE NOT EXISTS (SELECT info_hash FROM ingested_torrents WHERE ingested_torrents.info_hash = items.hash) AND (cat LIKE 'tv%' OR cat LIKE 'movies%');"
+FROM items where cat LIKE 'tv%%' ON CONFLICT DO NOTHING;"
+```
+
+And similarly for movies:
+```
+docker exec -it knightcrawler-postgres-1 psql -d knightcrawler -c "
+INSERT INTO ingested_torrents (name, source, category, info_hash, size, seeders, leechers, imdb, processed, \"createdAt\", \"updatedAt\")
+SELECT title, 'RARBG', 'movies', hash, size, NULL, NULL, imdb, false, current_timestamp, current_timestamp
+FROM items where cat LIKE 'movies%%' ON CONFLICT DO NOTHING;"
 ```
 
 You should get a response similar to:
@@ -227,6 +237,14 @@ After, you can delete the `items` table by running:
 ```docker exec -it knightcrawler-postgres-1 psql -d knightcrawler -c "drop table items";```
 
 
+#### Fixing imported databases
+
+If you've already imported a database with incorrect categories, you can fix these categories by running the following commands:
+
+```docker exec -it knightcrawler-postgres-1 psql -d knightcrawler -c "UPDATE ingested_torrents  SET category='movies', processed='f' WHERE category LIKE 'movies_%';"```
+
+```docker exec -it knightcrawler-postgres-1 psql -d knightcrawler -c "UPDATE ingested_torrents  SET category='movies', processed='f' WHERE category LIKE 'movies_%';"```
+
 ## Selfhostio to KnightCrawler Migration
 
 With the renaming of the project, you will have to change your database name in order to keep your existing data.