tools/knightcrawler
1
0
Fork 0
mirror of https://github.com/knightcrawler-stremio/knightcrawler.git synced 2024-12-20 03:29:51 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Big rewrite - distributed consumers for ingestion / scraping(scalable) - single producer written in c#.

Browse Source 
Changed from page scraping to rss xml scraping
Includes RealDebridManager hashlist decoding (requires a github readonly PAT as requests must be authenticated) - This allows ingestion of 200k+ entries in a few hours.
Simplifies a lot of torrentio to deal with new data
This commit is contained in:
iPromKnight
2024-02-01 16:38:45 +00:00
parent 6fb4ddcf23
commit ab17ef81be
255 changed files with 18489 additions and 69074 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
README.md
View File

@@ -6,4 +6,10 @@
```
docker-compose up -d
```
Then open your browser to `127.0.0.1:7000`
Then open your browser to `127.0.0.1:7000`
If you'd like to enable crawling of RealDebridManager's shared hashlists which will massively boost your database cached entries,
enter a readonly github personal access token in 'env/producer.env' as the 'GithubSettings__PAT=<token_here>' value.
You can scale the number of consumers, by changing the consumer deploy replica count in the compose file on line 87. This is currently set to 3.
If you'd like to adjust the number of concurrent processed ingestions per consumer, thats the job concurrency setting within 'env/consumer.env'.