brock/torrentio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,110 Commits 7 Branches 31 Tags
master
Commit Graph

1110 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
trulow
d4e82e9878 Updated and replaced selfhostio entries with knightcrawler 
Updated and replaced selfhostio entries with knightcrawler.

Was previously still pointing to the old DB name of selfhostio
 2024-02-04 16:52:05 -08:00
purple_emily
6e362cfb89 Merge pull request #62 from FunkeCoder23/hotfix-manifest-version 2024-02-04 22:30:15 +00:00
funkecoder23
3ddc922081 remove extraneous v from manifest version 2024-02-04 17:19:11 -05:00
iPromKnight
dae52c1bc4 forgot to update paths - doh 2024-02-04 20:32:08 +00:00
iPromKnight
b773ffcb20 Logging expansion 
Adds Promtail and Loki to the metrics stack, configuring them as a datasource in grafana on deployment
Adds a dashboard to monitor the system (logs for producer and consumer)
Restructures the base directory so deployment manifests are moved into a deployments folder
Adds selective labels to producer, consumer and addon so that their logs are ingested by promtail
 2024-02-04 19:50:28 +00:00
iPromKnight
eef3265850 Merge pull request #55 from Gabisonfire/critical-security-enhancements 
Critical security enhancements
 2024-02-04 16:04:42 +00:00
iPromKnight
2b715b90a7 Remove postgres exporter port bind 2024-02-04 16:02:47 +00:00
purple_emily
170d126fc9 Formatting 2024-02-04 16:02:47 +00:00
purple_emily
f4df6e27a7 Close the ports for RabbitMQ 2024-02-04 16:02:47 +00:00
purple_emily
562713f749 Close the MongoDB port 2024-02-04 16:02:23 +00:00
purple_emily
050c22e11e Change wording of security comment 2024-02-04 15:58:10 +00:00
purple_emily
bc2c36b92f Close the external PostgreSQL port 2024-02-04 15:58:10 +00:00
iPromKnight
875d79b44d Merge pull request #46 from purple-emily/change-to-single-env-file 
Simplify the environment variables
 2024-02-04 15:56:25 +00:00
iPromKnight
7ba38db0cc remove .env and add .env.example - add readme instructions 2024-02-04 15:45:32 +00:00
iPromKnight
e461e26b0f Change postgres configuration in the producer to use the env vars from the stack 2024-02-04 15:03:07 +00:00
purple_emily
c20a92e8f0 Remove old environment files 2024-02-04 08:34:49 +00:00
purple_emily
cd3c2d3fe6 Merge and simplify the environment variables in 
addon and consumer.

Todo: producer

Change DATABASE_URI to be generic POSTGRES variables

DOES NOT WORK - First pass at upgrading environment variables

PostgreSQL environment variables have been split for addon and consumer. ENABLE_SYNC hard coded as `true`

MongoDB variables update.

Make the addon code more similar to the consumer code

Get some parity between addon and consumer
 2024-02-04 08:30:55 +00:00
purple_emily
b7c3c4376b Merge pull request #51 from Gabisonfire/feat/renaming 
Renames project to Knight Crawler
 2024-02-04 08:30:24 +00:00
Gabisonfire
5fe3f471e9 Renames project to Knight Crawler 2024-02-03 23:45:06 -05:00
Gabisonfire
a8173a0ba7 Merge pull request #48 from HDwayne/master 
Monitoring with Grafana and Prometheus
 2024-02-03 19:42:49 -05:00
Gabisonfire
adcf911d10 Merge pull request #49 from iPromKnight/master 
Package updates and start of logging
 2024-02-03 19:38:52 -05:00
HDwayne
45e17799b2 Add monitoring for Postgres metrics in README 2024-02-04 01:29:55 +01:00
HDwayne
5f691ef877 Add Postgres exporter service 2024-02-04 01:18:34 +01:00
iPromKnight
d660a72d6f missed a couple of log lines for logger 2024-02-03 22:12:45 +00:00
iPromKnight
8c1b6fc91b roll logging levels out throughout the consumer 2024-02-03 22:06:38 +00:00
iPromKnight
4bd76d682f remove pm2 process manager - let the container orchestrator handle the lifetime events (docker, podman, k8s etc) 2024-02-03 22:06:02 +00:00
iPromKnight
678b3857f3 Add pino as a logger, and update cache manager 
Rewrites the cache service to use the latest implementation of cache manager, and bring in the new mongo package.
Introduce Logger service
Removes bluebird as a dependency
 2024-02-03 22:05:15 +00:00
HDwayne
a56b04f04b Add Temporary statistics endpoint 2024-02-03 22:06:13 +01:00
HDwayne
679a658463 Fix docker-compose command in README 2024-02-03 17:17:13 +01:00
HDwayne
8e991488ce Add monitoring capabilities with Grafana and Prometheus 2024-02-03 17:09:10 +01:00
Gabisonfire
f9746af9cd Merge pull request #47 from geek-cookbook/add-rarbg-docs 
Minimal docs on importing rarbg dump
 2024-02-03 07:43:00 -05:00
David Young
679bba3030 Minimal docs on importing rarbg dump 
Signed-off-by: David Young <davidy@funkypenguin.co.nz>
 2024-02-03 23:07:12 +13:00
Gabisonfire
cb0b22e6a9 Merge pull request #40 from purple-emily/update-the-documentation 
Update the documentation
 2024-02-02 11:01:30 -05:00
Gabisonfire
898ab6ed1b Merge pull request #26 from iPromKnight/master 
Big rewrite of scraping solution
 2024-02-02 11:00:34 -05:00
purple_emily
1a71d618bd Add a simple to do list 2024-02-02 15:40:12 +00:00
purple_emily
428bdff413 Added a section on how to generate a GitHub PAT 2024-02-02 15:37:01 +00:00
Gabisonfire
fd03760155 Add issues template 2024-02-02 10:13:00 -05:00
purple_emily
2ddd30f621 Add an overview section to the README 2024-02-02 15:04:00 +00:00
iPromKnight
57f4757541 Implement Max Queue and Max Batch size when publishing 
MaxPublishBatchSize must be set, but MaxQueueSize can be set to 0 to disable check of the rabbitmq queue size
 2024-02-02 14:43:29 +00:00
iPromKnight
8ad6cf731c harden consumer 2024-02-02 14:04:53 +00:00
iPromKnight
68edaba308 Introduce max batch size, and configurable publish window 
Still need to implement queue size limit
Also fixes env var consistency between addon and consumer
 2024-02-02 13:49:54 +00:00
iPromKnight
0644fe7720 add the jackett backend alternate addon "jackettio" 2024-02-02 13:27:41 +00:00
iPromKnight
188ffd10f3 rework removing providers filters, and clean up project a little 
Also wraps in pm2, and introduces linting, and dev watch
 2024-02-02 13:27:15 +00:00
iPromKnight
c1169a15ee Add editor config 2024-02-02 13:25:57 +00:00
iPromKnight
ee994fc8be ignore bin and obj 2024-02-01 16:47:45 +00:00
iPromKnight
ab17ef81be Big rewrite - distributed consumers for ingestion / scraping(scalable) - single producer written in c#. 
Changed from page scraping to rss xml scraping
Includes RealDebridManager hashlist decoding (requires a github readonly PAT as requests must be authenticated) - This allows ingestion of 200k+ entries in a few hours.
Simplifies a lot of torrentio to deal with new data
 2024-02-01 16:38:45 +00:00
Gabisonfire
6fb4ddcf23 Merge pull request #20 from Gabisonfire/devel 
Sync with upstream
 2024-01-31 20:29:43 -05:00
Gabisonfire
b2bba2a8aa Sync with upstream 2024-01-31 19:53:47 -05:00
Gabisonfire
d63f1d9716 Merge pull request #19 from KillTrot/master 
Change 1337x FlareSolverr mechanic to only use FlareSolverr on the first request
 2024-01-31 16:12:44 -05:00
Cedric
a0017f3491 Merge branch 'Gabisonfire:master' into master 2024-01-31 15:00:28 -06:00