purple_emily
cd3c2d3fe6
Merge and simplify the environment variables in
...
addon and consumer.
Todo: producer
Change DATABASE_URI to be generic POSTGRES variables
DOES NOT WORK - First pass at upgrading environment variables
PostgreSQL environment variables have been split for addon and consumer. ENABLE_SYNC hard coded as `true`
MongoDB variables update.
Make the addon code more similar to the consumer code
Get some parity between addon and consumer
2024-02-04 08:30:55 +00:00
purple_emily
b7c3c4376b
Merge pull request #51 from Gabisonfire/feat/renaming
...
Renames project to Knight Crawler
2024-02-04 08:30:24 +00:00
Gabisonfire
5fe3f471e9
Renames project to Knight Crawler
2024-02-03 23:45:06 -05:00
Gabisonfire
a8173a0ba7
Merge pull request #48 from HDwayne/master
...
Monitoring with Grafana and Prometheus
2024-02-03 19:42:49 -05:00
Gabisonfire
adcf911d10
Merge pull request #49 from iPromKnight/master
...
Package updates and start of logging
2024-02-03 19:38:52 -05:00
HDwayne
45e17799b2
Add monitoring for Postgres metrics in README
2024-02-04 01:29:55 +01:00
HDwayne
5f691ef877
Add Postgres exporter service
2024-02-04 01:18:34 +01:00
iPromKnight
d660a72d6f
missed a couple of log lines for logger
2024-02-03 22:12:45 +00:00
iPromKnight
8c1b6fc91b
roll logging levels out throughout the consumer
2024-02-03 22:06:38 +00:00
iPromKnight
4bd76d682f
remove pm2 process manager - let the container orchestrator handle the lifetime events (docker, podman, k8s etc)
2024-02-03 22:06:02 +00:00
iPromKnight
678b3857f3
Add pino as a logger, and update cache manager
...
Rewrites the cache service to use the latest implementation of cache manager, and bring in the new mongo package.
Introduce Logger service
Removes bluebird as a dependency
2024-02-03 22:05:15 +00:00
HDwayne
a56b04f04b
Add Temporary statistics endpoint
2024-02-03 22:06:13 +01:00
HDwayne
679a658463
Fix docker-compose command in README
2024-02-03 17:17:13 +01:00
HDwayne
8e991488ce
Add monitoring capabilities with Grafana and Prometheus
2024-02-03 17:09:10 +01:00
Gabisonfire
f9746af9cd
Merge pull request #47 from geek-cookbook/add-rarbg-docs
...
Minimal docs on importing rarbg dump
2024-02-03 07:43:00 -05:00
David Young
679bba3030
Minimal docs on importing rarbg dump
...
Signed-off-by: David Young <davidy@funkypenguin.co.nz >
2024-02-03 23:07:12 +13:00
Gabisonfire
cb0b22e6a9
Merge pull request #40 from purple-emily/update-the-documentation
...
Update the documentation
2024-02-02 11:01:30 -05:00
Gabisonfire
898ab6ed1b
Merge pull request #26 from iPromKnight/master
...
Big rewrite of scraping solution
2024-02-02 11:00:34 -05:00
purple_emily
1a71d618bd
Add a simple to do list
2024-02-02 15:40:12 +00:00
purple_emily
428bdff413
Added a section on how to generate a GitHub PAT
2024-02-02 15:37:01 +00:00
Gabisonfire
fd03760155
Add issues template
2024-02-02 10:13:00 -05:00
purple_emily
2ddd30f621
Add an overview section to the README
2024-02-02 15:04:00 +00:00
iPromKnight
57f4757541
Implement Max Queue and Max Batch size when publishing
...
MaxPublishBatchSize must be set, but MaxQueueSize can be set to 0 to disable check of the rabbitmq queue size
2024-02-02 14:43:29 +00:00
iPromKnight
8ad6cf731c
harden consumer
2024-02-02 14:04:53 +00:00
iPromKnight
68edaba308
Introduce max batch size, and configurable publish window
...
Still need to implement queue size limit
Also fixes env var consistency between addon and consumer
2024-02-02 13:49:54 +00:00
iPromKnight
0644fe7720
add the jackett backend alternate addon "jackettio"
2024-02-02 13:27:41 +00:00
iPromKnight
188ffd10f3
rework removing providers filters, and clean up project a little
...
Also wraps in pm2, and introduces linting, and dev watch
2024-02-02 13:27:15 +00:00
iPromKnight
c1169a15ee
Add editor config
2024-02-02 13:25:57 +00:00
iPromKnight
ee994fc8be
ignore bin and obj
2024-02-01 16:47:45 +00:00
iPromKnight
ab17ef81be
Big rewrite - distributed consumers for ingestion / scraping(scalable) - single producer written in c#.
...
Changed from page scraping to rss xml scraping
Includes RealDebridManager hashlist decoding (requires a github readonly PAT as requests must be authenticated) - This allows ingestion of 200k+ entries in a few hours.
Simplifies a lot of torrentio to deal with new data
2024-02-01 16:38:45 +00:00
Gabisonfire
6fb4ddcf23
Merge pull request #20 from Gabisonfire/devel
...
Sync with upstream
2024-01-31 20:29:43 -05:00
Gabisonfire
b2bba2a8aa
Sync with upstream
2024-01-31 19:53:47 -05:00
Gabisonfire
d63f1d9716
Merge pull request #19 from KillTrot/master
...
Change 1337x FlareSolverr mechanic to only use FlareSolverr on the first request
2024-01-31 16:12:44 -05:00
Cedric
a0017f3491
Merge branch 'Gabisonfire:master' into master
2024-01-31 15:00:28 -06:00
Cedric
3f6c95b12f
Remove FlareSolverr Variables after scraping
2024-01-31 14:59:41 -06:00
Cedric
aa101758f0
Merge branch 'master' of https://github.com/KillTrot/torrentio-scraper-sh
2024-01-31 14:21:48 -06:00
Cedric
ee34b6db17
Change Flaresolverr mechanic to use cookies after initial flaresolverr request
2024-01-31 14:19:46 -06:00
TheBeastLT
fb267b8057
remove psql logging
2024-01-31 09:30:59 +02:00
TheBeastLT
79b83e52e4
reduce psql connection pool a bit
2024-01-31 09:25:33 +02:00
TheBeastLT
ebf2934c6c
cache resolve url in remote mongo and reduce cache time
2024-01-31 09:24:31 +02:00
TheBeastLT
11e86c3cdc
increase RD timeout
2024-01-30 09:19:57 +02:00
TheBeastLT
aaa48f585c
update deploy workflow
2024-01-30 09:04:57 +02:00
TheBeastLT
fb8148a737
update deploy workflow
2024-01-30 08:58:45 +02:00
TheBeastLT
cca0331caa
remove bottleneck
2024-01-30 08:56:27 +02:00
TheBeastLT
85d9f3d406
update deploy workflow
2024-01-30 08:55:17 +02:00
TheBeastLT
ae955e6634
update deploy workflow
2024-01-30 08:50:12 +02:00
TheBeastLT
cc0d8cbdab
bump node base image to 21
2024-01-30 08:49:58 +02:00
Gabisonfire
3556821253
Merge pull request #15 from KillTrot/master
...
Add FlareSolverr fro 1337x
2024-01-29 10:49:30 -05:00
Cedric
21e8b628ef
Update docker-compose.yml
2024-01-29 09:16:50 -06:00
Cedric
07d4d020e6
Added FlareSolverr fro 1337x
2024-01-29 09:15:46 -06:00
