tools/knightcrawler
1
0
Fork 0
mirror of https://github.com/knightcrawler-stremio/knightcrawler.git synced 2024-12-20 03:29:51 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Big rewrite - distributed consumers for ingestion / scraping(scalable) - single producer written in c#.

Browse Source 
Changed from page scraping to rss xml scraping
Includes RealDebridManager hashlist decoding (requires a github readonly PAT as requests must be authenticated) - This allows ingestion of 200k+ entries in a few hours.
Simplifies a lot of torrentio to deal with new data
This commit is contained in:
iPromKnight
2024-02-01 16:38:45 +00:00
parent 6fb4ddcf23
commit ab17ef81be
255 changed files with 18489 additions and 69074 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
src/producer/GlobalUsings.cs Normal file
View File

@@ -0,0 +1,22 @@
// Global using directives
global using System.Text;
global using System.Text.Json;
global using System.Text.RegularExpressions;
global using System.Xml.Linq;
global using Dapper;
global using LZStringCSharp;
global using MassTransit;
global using Microsoft.AspNetCore.Builder;
global using Microsoft.Extensions.DependencyInjection;
global using Microsoft.Extensions.Logging;
global using Npgsql;
global using Quartz;
global using Scraper.Crawlers;
global using Scraper.Crawlers.Sites;
global using Scraper.Extensions;
global using Scraper.Interfaces;
global using Scraper.Jobs;
global using Scraper.Models;
global using Scraper.Services;
global using Serilog;