iPromKnight
1b9a01c677
BREAKING: Cleanup RabbitMQ env vars, and Github Pat
2024-02-28 12:57:55 +00:00
Gabisonfire
6c4282b6de
Adds Nyaa Crawler
2024-02-27 10:08:39 -05:00
iPromKnight
49a6283f26
Fix DMM so that all pages are enumerated
...
Fixes #95 by switching to git trees instead of the content api.
2024-02-27 13:51:21 +00:00
David Howell
2cae5296a2
Build multi-platform images
...
Refactor GitHub Actions workflow for build
Run Dockle and Trivy, upload sarif reports to GitHub
Refactor Dockerfiles based on best practices
2024-02-08 06:00:48 +00:00
iPromKnight
e461e26b0f
Change postgres configuration in the producer to use the env vars from the stack
2024-02-04 15:03:07 +00:00
iPromKnight
57f4757541
Implement Max Queue and Max Batch size when publishing
...
MaxPublishBatchSize must be set, but MaxQueueSize can be set to 0 to disable check of the rabbitmq queue size
2024-02-02 14:43:29 +00:00
iPromKnight
68edaba308
Introduce max batch size, and configurable publish window
...
Still need to implement queue size limit
Also fixes env var consistency between addon and consumer
2024-02-02 13:49:54 +00:00
iPromKnight
ee994fc8be
ignore bin and obj
2024-02-01 16:47:45 +00:00
iPromKnight
ab17ef81be
Big rewrite - distributed consumers for ingestion / scraping(scalable) - single producer written in c#.
...
Changed from page scraping to rss xml scraping
Includes RealDebridManager hashlist decoding (requires a github readonly PAT as requests must be authenticated) - This allows ingestion of 200k+ entries in a few hours.
Simplifies a lot of torrentio to deal with new data
2024-02-01 16:38:45 +00:00
