15 Commits

Author	SHA1	Message	Date
iPromKnight	c3a281c39f	retry polic and circuit breaker policy	2024-03-03 19:54:32 +00:00
iPromKnight	62decbf994	Ensure we throw ... when torrentio/knightcrawler instances return invalid status codes on fetch requests for json payloads, pre-parsing of json, polly will catch in the policy wrapped resiliency handler	2024-03-03 19:30:06 +00:00
iPromKnight	c61e9e94e1	Rethrow so polly captures failures on requests.	2024-03-03 19:22:10 +00:00
iPromKnight	d8f48fcee9	Introduce a circuit breaker, also exit out of loop if mongo failures.	2024-03-03 16:19:56 +00:00
iPromKnight	4b3bb2b5bd	hotfix continue, not break - add slight delay - log params	2024-03-03 04:10:14 +00:00
iPromKnight	95fa48c851	Woke up to see a discussion about torrentio scraping: powered by community ... Was a little inspired. Now we have a database (self populating) of imdb id's - why shouldn't we actually have the ability to scrape any other instance of torrentio, or knightcrawler? Also restructured the producer to be vertically sliced to make it easier to work with Too much flicking back and forth between Jobs and Crawlers when configuring	2024-03-02 18:41:57 +00:00
iPromKnight	1b9a01c677	BREAKING: Cleanup RabbitMQ env vars, and Github Pat	2024-02-28 12:57:55 +00:00
Gabisonfire	6c4282b6de	Adds Nyaa Crawler	2024-02-27 10:08:39 -05:00
iPromKnight	49a6283f26	Fix DMM so that all pages are enumerated ... Fixes #95 by switching to git trees instead of the content api.	2024-02-27 13:51:21 +00:00
David Howell	2cae5296a2	Build multi-platform images ... Refactor GitHub Actions workflow for build Run Dockle and Trivy, upload sarif reports to GitHub Refactor Dockerfiles based on best practices	2024-02-08 06:00:48 +00:00
iPromKnight	e461e26b0f	Change postgres configuration in the producer to use the env vars from the stack	2024-02-04 15:03:07 +00:00
iPromKnight	57f4757541	Implement Max Queue and Max Batch size when publishing ... MaxPublishBatchSize must be set, but MaxQueueSize can be set to 0 to disable check of the rabbitmq queue size	2024-02-02 14:43:29 +00:00
iPromKnight	68edaba308	Introduce max batch size, and configurable publish window ... Still need to implement queue size limit Also fixes env var consistency between addon and consumer	2024-02-02 13:49:54 +00:00
iPromKnight	ee994fc8be	ignore bin and obj	2024-02-01 16:47:45 +00:00
iPromKnight	ab17ef81be	Big rewrite - distributed consumers for ingestion / scraping(scalable) - single producer written in c#. ... Changed from page scraping to rss xml scraping Includes RealDebridManager hashlist decoding (requires a github readonly PAT as requests must be authenticated) - This allows ingestion of 200k+ entries in a few hours. Simplifies a lot of torrentio to deal with new data	2024-02-01 16:38:45 +00:00