Commit Graph

1121 Commits

Author SHA1 Message Date
iPromKnight
c3e58e4234 Fix redis connection strings for consistency across languages. (#168)
* Fix redis connection strings across languages

* compose version bump
v2.0.2
2024-03-26 09:26:35 +00:00
iPromKnight
d584102d60 image updates for patched release (#167) v2.0.1 2024-03-26 00:27:54 +00:00
iPromKnight
fe4bb59502 fix indenting on env file (#166)
* fix images :/

* fix indenting
2024-03-26 00:22:33 +00:00
iPromKnight
472b3342d5 fix images :/ (#165) 2024-03-26 00:01:59 +00:00
iPromKnight
b035ef596b change tag glob (#164) v2.0.0 2024-03-25 23:41:58 +00:00
iPromKnight
9a831e92d0 Producer / Consumer / Collector rewrite (#160)
* Converted metadata service to redis

* move to postgres instead

* fix global usings

* [skip ci] optimize wolverine by prebuilding static types

* [skip ci] Stop indexing mac folder indexes

* [skip ci] producer, metadata and migrations

removed mongodb
added redis cache
imdb meta in postgres
Enable pgtrm
Create trigrams index
Add search meta postgres function

* [skip ci] get rid of node folder, replace mongo with redis in consumer

also wire up postgres metadata searches

* [skip ci] change mongo to redis in the addon

* [skip ci] jackettio to redis

* Rest of mongo removed...

* Cleaner rerunning of metadata - without conflicts

* Add akas import as well as basic metadata

* Include episodes file too

* cascade truncate pre-import

* reverse order to avoid cascadeing

* separate out clean to separate handler

* Switch producer to use metadata matching pre-preocessing dmm

* More work

* Still porting PTN

* PTN port, adding tests

* [skip ci] Codec tests

* [skip ci] Complete Collection handler tests

* [skip ci] container tests

* [skip ci] Convert handlers tests

* [skip ci] DateHandler tests

* [skip ci] Dual Audio matching tests

* [skip ci] episode code tests

* [skip ci] Extended handler tests

* [skip ci] group handler tests

* [skip ci] some broken stuff right now

* [skip ci] more ptn

* [skip ci] PTN now in a separate nuget package, rebased this on the redis changes - i need them.

* [skip ci] Wire up PTN port. Tired - will test tomorrow

* [skip ci] Needs a lot of work - too many titles being missed now

* cleaner. done?

* Handle the date in the imdb search

- add integer function to confirm its a valid integer
- use the input date as a range of -+1 year

* [skip ci] Start of collector service for RD

[skip ci] WIP

Implemented metadata saga, along with channels to process up to a maximum of 100 infohashes each time
The saga will rety for each infohas by requeuing up to three times, before just marking as complete for that infoHash - meaning no data will be updated in the db for that torrent.

[skip ci] Ready to test with queue publishing

Will provision a fanout exchange if it doesn't exist, and create and bind a queue to it. Listens to the queue with 50 prefetch count.
Still needs PTN rewrite bringing in to parse the filename response from real debrid, and extract season and episode numbers if the file is a tvshow

[skip ci] Add Debrid Collector Build Job

Debrid Collector ready for testing

New consumer, new collector, producer has meta lookup and anti porn measures

[skip ci] WIP - moving from wolverine to MassTransit.

 not happy that wolverine cannot effectively control saga concurrency. we need to really.

[skip ci] Producer and new Consumer moved to MassTransit

Just the debrid collector to go now, then to write the optional qbit collector.

Collector now switched to mass transit too

hide porn titles in logs, clean up cache name in redis for imdb titles

[skip ci] Allow control of queues

[skip ci] Update deployment

Remove old consumer, fix deployment files, fix dockerfiles for shared project import

fix base deployment

* Add collector missing env var

* edits to kick off builds

* Add optional qbit deployment which qbit collector will use

* Qbit collector done

* reorder compose, and bring both qbit and qbitcollector into the compose, with 0 replicas as default

* Clean up compose file

* Ensure debrid collector errors if no debrid api key
2024-03-25 23:32:28 +00:00
iPromKnight
9c6c1ac249 Update compose versions 1.0.1, ready for tag push (#163) v1.0.1 2024-03-25 20:38:11 +00:00
iPromKnight
0ddfac57f7 Build on Tag Pushes. (#162)
* enable tag and pr builds

* Build on Tag Pushes
v1.0.0
2024-03-25 20:27:37 +00:00
iPromKnight
9fbd750cd2 enable tag and pr builds (#161) 2024-03-25 20:24:14 +00:00
Knight Crawler
5fc2027cfa Option to manually trigger each workflow (#159)
Co-authored-by: purple_emily <purple_emily@protonmail.com>
2024-03-20 20:26:32 +00:00
purple_emily
2d39476c65 Push dev builds & ready to tag semver (#153) 2024-03-14 14:27:19 +00:00
iPromKnight
e7f987a0d7 Merge pull request #151 from Gabisonfire/feature/tissue-corn-sanitizer
Improve producer matching - Add tissue service
unreleased-original-source
2024-03-12 10:31:18 +00:00
iPromKnight
79a6aa3cb0 Improve producer matching - Add tissue service
Tissue service will sanitize the existign database of ingested torrents by matching existing titles with new banned word lists. Now with added kleenex
2024-03-12 10:29:13 +00:00
iPromKnight
e24d81dd96 Merge pull request #149 from Gabisonfire/improve-consumer
Simplification of parsing in consumer
2024-03-11 11:00:23 +00:00
iPromKnight
aeb83c19f8 Simplification of parsing in consumer
should speed up massively especially if imdbIds are found from mongodb
2024-03-11 10:56:04 +00:00
iPromKnight
e23ee974e2 Merge pull request #148 from Gabisonfire/hotfix/nyaa
Fix nyaa category
2024-03-11 09:03:22 +00:00
iPromKnight
5c310427b4 Fix nyaa category 2024-03-11 08:59:55 +00:00
iPromKnight
b3d9be0b7a Merge pull request #147 from Gabisonfire/force_build
accidentally skipped build on last pr
2024-03-10 23:37:28 +00:00
iPromKnight
dda81ec5bf accidentally skipped build on last pr
tired..
2024-03-10 23:37:16 +00:00
iPromKnight
8eae288f10 Merge pull request #146 from Gabisonfire/hotfix/default_season_1
[skip ci] Final hotfix
2024-03-10 23:34:41 +00:00
iPromKnight
75ac89489e [skip ci] Final hotfix 2024-03-10 23:34:35 +00:00
iPromKnight
fa27b0cda9 Merge pull request #145 from Gabisonfire/hotfix/series_consumer
Fix series parsing
2024-03-10 22:28:00 +00:00
iPromKnight
500dd0d725 patch type 2024-03-10 22:28:06 +00:00
iPromKnight
6f4bc10f5a Fix series parsing 2024-03-10 21:38:55 +00:00
iPromKnight
1b3c190ed1 Merge pull request #144 from Gabisonfire/reduce_cpu_cycles
reduce cpu cycles in parsing in producer
2024-03-10 15:14:37 +00:00
iPromKnight
02150482df reduce cpu cycles in parsing in producer 2024-03-10 15:14:17 +00:00
iPromKnight
f18cd5b1ac Merge pull request #143 from Gabisonfire/extra_terms
Few extra terms getting through
2024-03-10 14:54:58 +00:00
iPromKnight
2e774058ff Few extra terms getting through 2024-03-10 14:54:25 +00:00
iPromKnight
4e84d7c9c3 Merge pull request #142 from Gabisonfire/feature/dmm-improvements
remove log line of adult content
2024-03-10 13:55:20 +00:00
iPromKnight
ad04d323b4 remove log line of adult content 2024-03-10 13:54:35 +00:00
iPromKnight
7d0b779bc8 Merge pull request #129 from Gabisonfire/feature/dmm-improvements
Improvements for DMM
2024-03-10 13:52:53 +00:00
iPromKnight
e2b45e799d [skip ci] Remove Debug logged adult terms found 2024-03-10 13:49:51 +00:00
iPromKnight
6c03f79933 Complete 2024-03-10 13:48:27 +00:00
iPromKnight
c8a1ebd8ae Bump large file to 2500kb because of Jav list.
Doesn't make sense to enable lfs for this file.
2024-03-10 13:48:14 +00:00
iPromKnight
320fccc8e8 [skip ci] More work on parsing - seasons to fix still and use banned words 2024-03-10 12:48:19 +00:00
iPromKnight
51246ed352 Ignore producer data dir from codespell hook 2024-03-10 12:48:19 +00:00
iPromKnight
8d82a17876 re-disable services other than dmm while developing
re-enable

disable again - will squash dont worry

enable again

disable again
2024-03-10 12:48:19 +00:00
iPromKnight
f719520b3b [skip ci] Ignore all run profiles to prevent pat leaking
reenable these, testing only producer should build
2024-03-10 12:48:19 +00:00
iPromKnight
bacb50e060 [skip ci] remove extra package no longer in use 2024-03-10 12:48:19 +00:00
iPromKnight
6600fceb1a Wip Blacklisting dmm porn
Create adult text classifier ML Model

wip - starting to write PTN in c#

More work on season, show and movie parsing

Remove ML project
2024-03-10 12:48:16 +00:00
purple_emily
5aba05f2b4 Merge pull request #141 from Gabisonfire/generic-fixes
Typo at the end of the staging environment
2024-03-10 12:45:58 +00:00
purple_emily
601dbdf64f Typo at the end of the staging environment 2024-03-10 12:27:21 +00:00
iPromKnight
ddc038974e Merge pull request #139 from Gabisonfire/feature/migrator
Initial Database migrations service
2024-03-10 02:19:16 +00:00
iPromKnight
0b3000af85 Merge pull request #127 from Gabisonfire/move-optional-services
Move optional services
2024-03-10 00:46:54 +00:00
iPromKnight
b7cc35b60c add more coverage of tools and languages to git ignore
added gitkeep back in
2024-03-10 00:43:50 +00:00
purple_emily
64cf478725 Reorganise promtail to be alphabetical 2024-03-10 00:43:50 +00:00
purple_emily
c2f94d9f4e Missed a .env one-liner 2024-03-10 00:43:50 +00:00
purple_emily
28e6c09612 Rename to .yaml to keep consistency 2024-03-10 00:43:50 +00:00
purple_emily
d8885b23bc Simple formatting preference 2024-03-10 00:43:50 +00:00
purple_emily
cc6914cb86 Fix .env mapping 2024-03-10 00:43:50 +00:00