HDwayne
8e991488ce
Add monitoring capabilities with Grafana and Prometheus
2024-02-03 17:09:10 +01:00
Gabisonfire
f9746af9cd
Merge pull request #47 from geek-cookbook/add-rarbg-docs
...
Minimal docs on importing rarbg dump
2024-02-03 07:43:00 -05:00
David Young
679bba3030
Minimal docs on importing rarbg dump
...
Signed-off-by: David Young <davidy@funkypenguin.co.nz >
2024-02-03 23:07:12 +13:00
Gabisonfire
cb0b22e6a9
Merge pull request #40 from purple-emily/update-the-documentation
...
Update the documentation
2024-02-02 11:01:30 -05:00
Gabisonfire
898ab6ed1b
Merge pull request #26 from iPromKnight/master
...
Big rewrite of scraping solution
2024-02-02 11:00:34 -05:00
purple_emily
1a71d618bd
Add a simple to do list
2024-02-02 15:40:12 +00:00
purple_emily
428bdff413
Added a section on how to generate a GitHub PAT
2024-02-02 15:37:01 +00:00
Gabisonfire
fd03760155
Add issues template
2024-02-02 10:13:00 -05:00
purple_emily
2ddd30f621
Add an overview section to the README
2024-02-02 15:04:00 +00:00
iPromKnight
57f4757541
Implement Max Queue and Max Batch size when publishing
...
MaxPublishBatchSize must be set, but MaxQueueSize can be set to 0 to disable check of the rabbitmq queue size
2024-02-02 14:43:29 +00:00
iPromKnight
8ad6cf731c
harden consumer
2024-02-02 14:04:53 +00:00
iPromKnight
68edaba308
Introduce max batch size, and configurable publish window
...
Still need to implement queue size limit
Also fixes env var consistency between addon and consumer
2024-02-02 13:49:54 +00:00
iPromKnight
0644fe7720
add the jackett backend alternate addon "jackettio"
2024-02-02 13:27:41 +00:00
iPromKnight
188ffd10f3
rework removing providers filters, and clean up project a little
...
Also wraps in pm2, and introduces linting, and dev watch
2024-02-02 13:27:15 +00:00
iPromKnight
c1169a15ee
Add editor config
2024-02-02 13:25:57 +00:00
iPromKnight
ee994fc8be
ignore bin and obj
2024-02-01 16:47:45 +00:00
iPromKnight
ab17ef81be
Big rewrite - distributed consumers for ingestion / scraping(scalable) - single producer written in c#.
...
Changed from page scraping to rss xml scraping
Includes RealDebridManager hashlist decoding (requires a github readonly PAT as requests must be authenticated) - This allows ingestion of 200k+ entries in a few hours.
Simplifies a lot of torrentio to deal with new data
2024-02-01 16:38:45 +00:00
Gabisonfire
6fb4ddcf23
Merge pull request #20 from Gabisonfire/devel
...
Sync with upstream
2024-01-31 20:29:43 -05:00
Gabisonfire
b2bba2a8aa
Sync with upstream
2024-01-31 19:53:47 -05:00
Gabisonfire
d63f1d9716
Merge pull request #19 from KillTrot/master
...
Change 1337x FlareSolverr mechanic to only use FlareSolverr on the first request
2024-01-31 16:12:44 -05:00
Cedric
a0017f3491
Merge branch 'Gabisonfire:master' into master
2024-01-31 15:00:28 -06:00
Cedric
3f6c95b12f
Remove FlareSolverr Variables after scraping
2024-01-31 14:59:41 -06:00
Cedric
aa101758f0
Merge branch 'master' of https://github.com/KillTrot/torrentio-scraper-sh
2024-01-31 14:21:48 -06:00
Cedric
ee34b6db17
Change Flaresolverr mechanic to use cookies after initial flaresolverr request
2024-01-31 14:19:46 -06:00
TheBeastLT
fb267b8057
remove psql logging
2024-01-31 09:30:59 +02:00
TheBeastLT
79b83e52e4
reduce psql connection pool a bit
2024-01-31 09:25:33 +02:00
TheBeastLT
ebf2934c6c
cache resolve url in remote mongo and reduce cache time
2024-01-31 09:24:31 +02:00
TheBeastLT
11e86c3cdc
increase RD timeout
2024-01-30 09:19:57 +02:00
TheBeastLT
aaa48f585c
update deploy workflow
2024-01-30 09:04:57 +02:00
TheBeastLT
fb8148a737
update deploy workflow
2024-01-30 08:58:45 +02:00
TheBeastLT
cca0331caa
remove bottleneck
2024-01-30 08:56:27 +02:00
TheBeastLT
85d9f3d406
update deploy workflow
2024-01-30 08:55:17 +02:00
TheBeastLT
ae955e6634
update deploy workflow
2024-01-30 08:50:12 +02:00
TheBeastLT
cc0d8cbdab
bump node base image to 21
2024-01-30 08:49:58 +02:00
Gabisonfire
3556821253
Merge pull request #15 from KillTrot/master
...
Add FlareSolverr fro 1337x
2024-01-29 10:49:30 -05:00
Cedric
21e8b628ef
Update docker-compose.yml
2024-01-29 09:16:50 -06:00
Cedric
07d4d020e6
Added FlareSolverr fro 1337x
2024-01-29 09:15:46 -06:00
Gabisonfire
85af1bb66d
Merge pull request #10 from trulow/master
...
Added depends on to docker-compose.yml
2024-01-28 16:37:12 -05:00
TheBeastLT
81218bbc8d
increase mongo connection pool
2024-01-28 21:44:54 +02:00
TheBeastLT
3bf13e68ba
try use p-limit instead of bottleneck
2024-01-28 02:12:41 +02:00
trulow
ebd97eb250
Added depends on to docker-compose.yml
...
Added the following lines to both scraper and torrentio services
depends_on:
- mongodb
- postgres
2024-01-26 11:05:12 -08:00
Gabisonfire
7bddd41fac
Merge branch 'TheBeastLT:master' into master
2024-01-25 21:27:58 -05:00
TheBeastLT
64b8a55f3b
do not limit stream request named queue
2024-01-26 01:30:25 +02:00
TheBeastLT
9366ad1ccb
fix request named queue
2024-01-26 01:20:35 +02:00
TheBeastLT
4681cb1f1b
add named queue for stream requests
2024-01-26 01:12:44 +02:00
TheBeastLT
af7511a610
export named queue creation
2024-01-26 01:12:02 +02:00
TheBeastLT
0a06018f83
reduce limiter concurrent
2024-01-26 00:27:17 +02:00
TheBeastLT
3f1624cd8e
add temp logging
2024-01-26 00:14:52 +02:00
TheBeastLT
a71a846dbd
bring back sw stats
2024-01-26 00:09:18 +02:00
TheBeastLT
a22d7d4b10
improve psql pool config
2024-01-22 21:58:53 +02:00
