42 Commits

Author SHA1 Message Date
iPromKnight
baec0450bf Hotfix ingestor github flow, and move to top level src folder (foldedr per service) (#179) 2024-03-28 10:20:26 +00:00
RohirrimRider
cc15a69517 fix torrent_ingestor ci (#178) 2024-03-27 21:38:59 -05:00
RohirrimRider
a6d3a4a066 init ingest torrents from annatar (#157)
* init ingest torrents from annatar

* works

* mv annatar to src/

* done

* add ci and readme

---------

Co-authored-by: Brett <eruiluvatar@pnbx.xyz>
2024-03-27 21:35:03 -05:00
iPromKnight
b035ef596b change tag glob (#164) 2024-03-25 23:41:58 +00:00
iPromKnight
9a831e92d0 Producer / Consumer / Collector rewrite (#160)
* Converted metadata service to redis

* move to postgres instead

* fix global usings

* [skip ci] optimize wolverine by prebuilding static types

* [skip ci] Stop indexing mac folder indexes

* [skip ci] producer, metadata and migrations

removed mongodb
added redis cache
imdb meta in postgres
Enable pgtrm
Create trigrams index
Add search meta postgres function

* [skip ci] get rid of node folder, replace mongo with redis in consumer

also wire up postgres metadata searches

* [skip ci] change mongo to redis in the addon

* [skip ci] jackettio to redis

* Rest of mongo removed...

* Cleaner rerunning of metadata - without conflicts

* Add akas import as well as basic metadata

* Include episodes file too

* cascade truncate pre-import

* reverse order to avoid cascadeing

* separate out clean to separate handler

* Switch producer to use metadata matching pre-preocessing dmm

* More work

* Still porting PTN

* PTN port, adding tests

* [skip ci] Codec tests

* [skip ci] Complete Collection handler tests

* [skip ci] container tests

* [skip ci] Convert handlers tests

* [skip ci] DateHandler tests

* [skip ci] Dual Audio matching tests

* [skip ci] episode code tests

* [skip ci] Extended handler tests

* [skip ci] group handler tests

* [skip ci] some broken stuff right now

* [skip ci] more ptn

* [skip ci] PTN now in a separate nuget package, rebased this on the redis changes - i need them.

* [skip ci] Wire up PTN port. Tired - will test tomorrow

* [skip ci] Needs a lot of work - too many titles being missed now

* cleaner. done?

* Handle the date in the imdb search

- add integer function to confirm its a valid integer
- use the input date as a range of -+1 year

* [skip ci] Start of collector service for RD

[skip ci] WIP

Implemented metadata saga, along with channels to process up to a maximum of 100 infohashes each time
The saga will rety for each infohas by requeuing up to three times, before just marking as complete for that infoHash - meaning no data will be updated in the db for that torrent.

[skip ci] Ready to test with queue publishing

Will provision a fanout exchange if it doesn't exist, and create and bind a queue to it. Listens to the queue with 50 prefetch count.
Still needs PTN rewrite bringing in to parse the filename response from real debrid, and extract season and episode numbers if the file is a tvshow

[skip ci] Add Debrid Collector Build Job

Debrid Collector ready for testing

New consumer, new collector, producer has meta lookup and anti porn measures

[skip ci] WIP - moving from wolverine to MassTransit.

 not happy that wolverine cannot effectively control saga concurrency. we need to really.

[skip ci] Producer and new Consumer moved to MassTransit

Just the debrid collector to go now, then to write the optional qbit collector.

Collector now switched to mass transit too

hide porn titles in logs, clean up cache name in redis for imdb titles

[skip ci] Allow control of queues

[skip ci] Update deployment

Remove old consumer, fix deployment files, fix dockerfiles for shared project import

fix base deployment

* Add collector missing env var

* edits to kick off builds

* Add optional qbit deployment which qbit collector will use

* Qbit collector done

* reorder compose, and bring both qbit and qbitcollector into the compose, with 0 replicas as default

* Clean up compose file

* Ensure debrid collector errors if no debrid api key
2024-03-25 23:32:28 +00:00
iPromKnight
0ddfac57f7 Build on Tag Pushes. (#162)
* enable tag and pr builds

* Build on Tag Pushes
2024-03-25 20:27:37 +00:00
iPromKnight
9fbd750cd2 enable tag and pr builds (#161) 2024-03-25 20:24:14 +00:00
Knight Crawler
5fc2027cfa Option to manually trigger each workflow (#159)
Co-authored-by: purple_emily <purple_emily@protonmail.com>
2024-03-20 20:26:32 +00:00
purple_emily
2d39476c65 Push dev builds & ready to tag semver (#153) 2024-03-14 14:27:19 +00:00
iPromKnight
79a6aa3cb0 Improve producer matching - Add tissue service
Tissue service will sanitize the existign database of ingested torrents by matching existing titles with new banned word lists. Now with added kleenex
2024-03-12 10:29:13 +00:00
iPromKnight
b0c3675f36 add migrator build job 2024-03-09 19:54:13 +00:00
iPromKnight
09a8407a02 Use the env up front and pass to output maybe? 2024-03-09 16:58:01 +00:00
iPromKnight
d2ca5a85c5 try setting to inputs directly, not sure on the syntax on this 2024-03-09 16:52:02 +00:00
iPromKnight
a1b9f7d4f4 explicitly set the outputs 2024-03-09 16:42:37 +00:00
iPromKnight
cef44d2600 see if set vars works 2024-03-09 16:34:05 +00:00
iPromKnight
5f617df425 Remove env usage in names of steps 2024-03-09 16:16:09 +00:00
iPromKnight
46e6058e76 remove matrix from base 2024-03-09 16:12:02 +00:00
iPromKnight
608f344f42 Hotfix the parameter passthrough 2024-03-09 16:03:36 +00:00
iPromKnight
c3c6a71da7 Fix names for individual workflows 2024-03-09 15:54:13 +00:00
iPromKnight
96e6fd8b9b Skip workflow to base, and then one for each docker image
Also limits each to firing based on the path of changes in the push.
2024-03-09 15:45:52 +00:00
purple_emily
79409915cf Run pre-commit 2024-03-08 14:34:53 +00:00
iPromKnight
e24fb69dc3 build metadata image 2024-02-28 10:31:39 +00:00
Sleeyax
434c4083e7 Try to fix GitHub action workflow (see #87) 2024-02-18 13:52:27 +01:00
David Howell
2cae5296a2 Build multi-platform images
Refactor GitHub Actions workflow for build
Run Dockle and Trivy, upload sarif reports to GitHub
Refactor Dockerfiles based on best practices
2024-02-08 06:00:48 +00:00
Gabisonfire
32dced5f72 Builds and pushes various components images 2024-02-04 21:49:29 -05:00
Gabisonfire
2c41657d25 Builds and pushes various components images 2024-02-04 21:35:16 -05:00
Gabisonfire
fd03760155 Add issues template 2024-02-02 10:13:00 -05:00
Gabisonfire
a34e40fbc8 First self-hosted release 2024-01-17 00:27:03 -05:00
TheBeastLT
a4cdccc469 add swagger stats 2023-10-31 18:39:32 +02:00
TheBeastLT
9990acf36e retrieve resolver host from request instead of static value 2023-10-31 11:23:30 +02:00
TheBeastLT
a253e8e0b0 remove scraper 2022-06-16 17:43:27 +03:00
TheBeastLT
7422d3039b fix catalogs build command 2022-04-01 14:22:21 +02:00
TheBeastLT
120f30257d fix catalogs workflow 2022-04-01 14:01:34 +02:00
TheBeastLT
7a3b1721d3 add catalogs deployment 2022-04-01 13:57:37 +02:00
TheBeastLT
47c05c465c add erai rss scraper 2022-02-19 13:04:47 +01:00
TheBeastLT
5bf61760ea enable addon pipeline 2021-09-20 18:45:48 +02:00
TheBeastLT
fb60e56564 improve ondebaixa api 2021-09-17 15:11:43 +02:00
TheBeastLT
ea9a22ce63 [scraper] adds scraper deploy pipeline 2020-09-11 19:43:17 +02:00
TheBeastLT
fed5098b88 [addon] moves rd resolver to addon 2020-04-08 21:44:15 +02:00
TheBeastLT
abe67cdf4c deploy only when there are changes in addon 2020-03-26 18:44:59 +01:00
TheBeastLT
6244208ed6 adds docker deployment 2020-03-26 18:34:29 +01:00
TheBeastLT
19052cc366 [scraper] deploy to google app engine 2020-03-13 22:23:31 +01:00