Compare commits

..

12 Commits

68 changed files with 3232 additions and 514 deletions

View File

@ -10,7 +10,23 @@
"Bash(/home/anthony/go/bin/go build *)", "Bash(/home/anthony/go/bin/go build *)",
"Bash(npm install *)", "Bash(npm install *)",
"Bash(/home/anthony/.config/JetBrains/WebStorm2026.1/node/versions/24.14.1/bin/npm install *)", "Bash(/home/anthony/.config/JetBrains/WebStorm2026.1/node/versions/24.14.1/bin/npm install *)",
"Bash(npm run *)" "Bash(npm run *)",
"Bash(go build *)",
"Bash(npx tsc *)",
"Bash(node_modules/.bin/tsc --noEmit)",
"Bash(/usr/bin/node node_modules/.bin/tsc --noEmit)",
"Bash(fish -c \"which node\")",
"Read(//opt/**)",
"Bash(/home/anthony/.config/JetBrains/WebStorm2026.1/node/versions/24.14.1/bin/node node_modules/.bin/tsc --noEmit)",
"Bash(chmod +x /home/anthony/Documents/Projects/Tradarr/build-push.sh)",
"Bash(fish -c \"npm install react-markdown\")",
"Bash(fish -c \"which npm; which pnpm; which bun\")",
"Bash(/usr/bin/npm run *)",
"Bash(/home/anthony/.config/JetBrains/WebStorm2026.1/node/versions/24.14.1/lib/node_modules/npm/bin/npm run *)",
"WebFetch(domain:docs.ollama.com)",
"WebFetch(domain:github.com)",
"Bash(docker compose *)",
"Bash(sudo docker *)"
] ]
} }
} }

3
.gitignore vendored
View File

@ -28,3 +28,6 @@ Thumbs.db
# Logs # Logs
*.log *.log
# Scripts de déploiement
build-push.sh

154
README.md
View File

@ -0,0 +1,154 @@
# Tradarr
Agrégateur d'actualités financières avec résumés IA personnalisés par utilisateur.
Tradarr scrape des sources d'information (Bloomberg, StockTwits, Reuters, WatcherGuru), croise les articles avec la watchlist de chaque trader, puis génère via IA un résumé structuré orienté trading. L'utilisateur peut ensuite sélectionner des extraits du résumé et poser des questions à l'IA pour approfondir une analyse.
## Fonctionnalités
- Scraping automatique de plusieurs sources financières (planifiable)
- Résumés IA personnalisés selon la watchlist de chaque utilisateur
- Filtrage intelligent des articles par pertinence (passe 1) avant résumé (passe 2)
- Questions/réponses IA sur des extraits de résumés (rapports)
- Support de plusieurs fournisseurs IA : OpenAI, Anthropic, Gemini, Ollama, Claude Code CLI
- Interface PWA (installable sur mobile)
- Panel d'administration complet
## Installation (production)
### Prérequis
- Docker et Docker Compose installés sur le serveur
- Si tu utilises **Claude Code** comme fournisseur IA : Claude Code CLI installé et authentifié sur la machine hôte (`claude login`)
### 1. Créer le fichier `.env`
```bash
cp .env.example .env
```
Édite `.env` avec tes valeurs :
```env
# PostgreSQL
POSTGRES_DB=tradarr
POSTGRES_USER=tradarr
POSTGRES_PASSWORD=<mot_de_passe_fort>
# Secret JWT pour les tokens de session
JWT_SECRET=<chaîne_aléatoire_longue>
# Clé de chiffrement AES-256 pour les clés API et credentials (32 bytes en hex)
# Générer avec : openssl rand -hex 32
ENCRYPTION_KEY=<32_bytes_en_hex>
# Compte administrateur créé automatiquement au premier démarrage
ADMIN_EMAIL=admin@example.com
ADMIN_PASSWORD=<mot_de_passe_fort>
# Port exposé sur le serveur (défaut : 80)
FRONTEND_PORT=80
```
### 2. Ajuster `docker-compose.prod.yml` si nécessaire
#### Fournisseur Claude Code CLI (optionnel)
Si tu veux utiliser **Claude Code** comme fournisseur IA, le backend doit accéder aux credentials de ta session Claude. La configuration par défaut monte `/home/anthony/.claude` — adapte ce chemin à l'utilisateur qui exécute Docker sur ton serveur :
```yaml
# dans docker-compose.prod.yml, section backend > volumes
volumes:
- /home/<ton_user>/.claude:/root/.claude
```
#### Images Docker
Les images sont référencées par tag dans `docker-compose.prod.yml`. Mets à jour les tags selon la version à déployer :
```yaml
backend:
image: gitea.anthonybouteiller.ovh/blomios/tradarr-backend:v1.0.0
frontend:
image: gitea.anthonybouteiller.ovh/blomios/tradarr-frontend:v1.0.0
scraper:
image: gitea.anthonybouteiller.ovh/blomios/tradarr-scraper:v1.0.0
```
### 3. Démarrer
```bash
docker compose -f docker-compose.prod.yml up -d
```
L'interface est accessible sur `http://<serveur>:<FRONTEND_PORT>`.
---
## Configuration post-démarrage (panel admin)
Connecte-toi avec le compte admin défini dans `.env`, puis accède à **Admin** pour configurer :
### Fournisseurs IA
Ajoute au moins un fournisseur IA et assigne-le aux trois rôles disponibles :
| Rôle | Description |
|------|-------------|
| Résumés | Génération du résumé quotidien (passe 2) |
| Rapports | Réponses aux questions sur les résumés |
| Filtre articles | Sélection des articles pertinents (passe 1) |
Fournisseurs supportés :
| Nom | Clé API requise | Endpoint |
|-----|----------------|----------|
| `openai` | Oui | — |
| `anthropic` | Oui | — |
| `gemini` | Oui | — |
| `ollama` | Non | `http://ollama:11434` (par défaut) |
| `claudecode` | Non | — (utilise le CLI local) |
### Paramètres
| Clé | Description | Défaut |
|-----|-------------|--------|
| `articles_lookback_hours` | Fenêtre de temps pour récupérer les articles | `24` |
| `summary_max_articles` | Nombre max d'articles envoyés au modèle pour le résumé | `50` |
| `filter_batch_size` | Taille des lots pour la passe de filtrage IA | `20` |
| `timezone` | Fuseau horaire affiché dans les résumés | `UTC` |
| `ai_system_prompt` | Prompt système pour la génération des résumés | prompt par défaut |
### Planning
Configure les créneaux de génération automatique des résumés (jours et heures).
### Sources
Active ou désactive les sources de scraping :
- **Bloomberg** — nécessite des credentials (onglet Credentials)
- **StockTwits** — public, aucune configuration requise
- **Reuters** — public
- **WatcherGuru** — public
### Credentials Bloomberg
Renseigne ton login/mot de passe Bloomberg pour activer le scraping de cette source.
---
## Développement local
```bash
# Démarrer tous les services
docker compose up -d
# Rebuild après modification du code
docker compose build backend frontend && docker compose up -d backend frontend
```
Logs :
```bash
docker compose logs -f backend
docker compose logs -f frontend
```

View File

@ -9,15 +9,18 @@ RUN CGO_ENABLED=0 GOOS=linux go build -o tradarr ./cmd/server
FROM debian:bookworm-slim FROM debian:bookworm-slim
# Chromium pour le scraping Bloomberg # Chromium pour le scraping Bloomberg + Node.js pour Claude Code CLI
RUN apt-get update && apt-get install -y \ RUN apt-get update && apt-get install -y \
chromium \ chromium \
chromium-driver \ chromium-driver \
ca-certificates \ ca-certificates \
fonts-liberation \ fonts-liberation \
libnss3 \ libnss3 \
nodejs \
npm \
--no-install-recommends && \ --no-install-recommends && \
rm -rf /var/lib/apt/lists/* rm -rf /var/lib/apt/lists/* && \
npm install -g @anthropic-ai/claude-code
WORKDIR /app WORKDIR /app
COPY --from=builder /app/tradarr . COPY --from=builder /app/tradarr .

View File

@ -15,6 +15,8 @@ import (
"github.com/tradarr/backend/internal/scheduler" "github.com/tradarr/backend/internal/scheduler"
"github.com/tradarr/backend/internal/scraper" "github.com/tradarr/backend/internal/scraper"
"github.com/tradarr/backend/internal/scraper/bloomberg" "github.com/tradarr/backend/internal/scraper/bloomberg"
"github.com/tradarr/backend/internal/scraper/reuters"
"github.com/tradarr/backend/internal/scraper/watcherguru"
"github.com/tradarr/backend/internal/scraper/yahoofinance" "github.com/tradarr/backend/internal/scraper/yahoofinance"
) )
@ -38,30 +40,23 @@ func main() {
enc := crypto.New(cfg.EncryptionKey) enc := crypto.New(cfg.EncryptionKey)
pipeline := ai.NewPipeline(repo, enc) pipeline := ai.NewPipeline(repo, enc)
// Créer le compte admin initial si nécessaire
if err := ensureAdmin(repo, cfg); err != nil { if err := ensureAdmin(repo, cfg); err != nil {
log.Printf("ensure admin: %v", err) log.Printf("ensure admin: %v", err)
} }
// Configurer les scrapers
registry := scraper.NewRegistry(repo) registry := scraper.NewRegistry(repo)
registry.Register(bloomberg.NewDynamic(repo, enc, cfg.ScraperURL))
registry.Register(yahoofinance.New())
registry.Register(reuters.New())
registry.Register(watcherguru.New())
// Bloomberg (credentials chargés depuis la DB à chaque run)
bbScraper := bloomberg.NewDynamic(repo, enc, cfg.ChromePath)
registry.Register(bbScraper)
stScraper := yahoofinance.New()
registry.Register(stScraper)
// Scheduler
sched := scheduler.New(registry, pipeline, repo) sched := scheduler.New(registry, pipeline, repo)
if err := sched.Start(); err != nil { if err := sched.Start(); err != nil {
log.Printf("scheduler: %v", err) log.Printf("scheduler: %v", err)
} }
defer sched.Stop() defer sched.Stop()
// API h := handlers.New(repo, cfg, enc, registry, pipeline, sched)
h := handlers.New(repo, cfg, enc, registry, pipeline)
r := api.SetupRouter(h, cfg.JWTSecret) r := api.SetupRouter(h, cfg.JWTSecret)
addr := fmt.Sprintf(":%s", cfg.Port) addr := fmt.Sprintf(":%s", cfg.Port)

View File

@ -28,7 +28,7 @@ func newAnthropic(apiKey, model string) *anthropicProvider {
func (p *anthropicProvider) Name() string { return "anthropic" } func (p *anthropicProvider) Name() string { return "anthropic" }
func (p *anthropicProvider) Summarize(ctx context.Context, prompt string) (string, error) { func (p *anthropicProvider) Summarize(ctx context.Context, prompt string, _ GenOptions) (string, error) {
body := map[string]interface{}{ body := map[string]interface{}{
"model": p.model, "model": p.model,
"max_tokens": 4096, "max_tokens": 4096,

View File

@ -0,0 +1,52 @@
package ai
import (
"bytes"
"context"
"fmt"
"os/exec"
"strings"
)
type claudeCodeProvider struct {
model string
}
func newClaudeCode(model string) *claudeCodeProvider {
if model == "" {
model = "claude-sonnet-4-6"
}
return &claudeCodeProvider{model: model}
}
func (p *claudeCodeProvider) Name() string { return "claudecode" }
func (p *claudeCodeProvider) Summarize(ctx context.Context, prompt string, _ GenOptions) (string, error) {
var stdout, stderr bytes.Buffer
cmd := exec.CommandContext(ctx, "claude", "-p", prompt, "--model", p.model)
cmd.Stdout = &stdout
cmd.Stderr = &stderr
err := cmd.Run()
fmt.Printf("[claudecode] stdout len=%d stderr=%q err=%v\n", stdout.Len(), stderr.String(), err)
if err != nil {
msg := strings.TrimSpace(stderr.String())
if msg == "" {
msg = strings.TrimSpace(stdout.String())
}
if msg == "" {
msg = err.Error()
}
return "", fmt.Errorf("claude cli: %s", msg)
}
return strings.TrimSpace(stdout.String()), nil
}
func (p *claudeCodeProvider) ListModels(_ context.Context) ([]string, error) {
return []string{
"claude-opus-4-7",
"claude-sonnet-4-6",
"claude-haiku-4-5-20251001",
}, nil
}

View File

@ -28,7 +28,7 @@ func newGemini(apiKey, model string) *geminiProvider {
func (p *geminiProvider) Name() string { return "gemini" } func (p *geminiProvider) Name() string { return "gemini" }
func (p *geminiProvider) Summarize(ctx context.Context, prompt string) (string, error) { func (p *geminiProvider) Summarize(ctx context.Context, prompt string, _ GenOptions) (string, error) {
url := fmt.Sprintf( url := fmt.Sprintf(
"https://generativelanguage.googleapis.com/v1beta/models/%s:generateContent?key=%s", "https://generativelanguage.googleapis.com/v1beta/models/%s:generateContent?key=%s",
p.model, p.apiKey, p.model, p.apiKey,
@ -74,11 +74,43 @@ func (p *geminiProvider) Summarize(ctx context.Context, prompt string) (string,
return result.Candidates[0].Content.Parts[0].Text, nil return result.Candidates[0].Content.Parts[0].Text, nil
} }
func (p *geminiProvider) ListModels(_ context.Context) ([]string, error) { func (p *geminiProvider) ListModels(ctx context.Context) ([]string, error) {
return []string{ url := fmt.Sprintf("https://generativelanguage.googleapis.com/v1beta/models?key=%s", p.apiKey)
"gemini-2.0-flash", req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
"gemini-2.0-flash-lite", if err != nil {
"gemini-1.5-pro", return nil, err
"gemini-1.5-flash", }
}, nil resp, err := p.client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
raw, _ := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("gemini list models error %d: %s", resp.StatusCode, raw)
}
var result struct {
Models []struct {
Name string `json:"name"`
SupportedMethods []string `json:"supportedGenerationMethods"`
} `json:"models"`
}
if err := json.Unmarshal(raw, &result); err != nil {
return nil, err
}
var names []string
for _, m := range result.Models {
for _, method := range m.SupportedMethods {
if method == "generateContent" {
// name is "models/gemini-xxx", strip prefix
id := m.Name
if len(id) > 7 {
id = id[7:] // strip "models/"
}
names = append(names, id)
break
}
}
}
return names, nil
} }

View File

@ -7,37 +7,62 @@ import (
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
"time"
) )
type ollamaProvider struct { // OllamaModelInfo holds detailed info about an installed Ollama model.
type OllamaModelInfo struct {
Name string `json:"name"`
Size int64 `json:"size"`
ModifiedAt string `json:"modified_at"`
Details struct {
ParameterSize string `json:"parameter_size"`
QuantizationLevel string `json:"quantization_level"`
Family string `json:"family"`
} `json:"details"`
}
// OllamaProvider implements Provider for Ollama and also exposes model management operations.
type OllamaProvider struct {
endpoint string endpoint string
model string model string
client *http.Client client *http.Client
} }
func newOllama(endpoint, model string) *ollamaProvider { func newOllama(endpoint, model string) *OllamaProvider {
if endpoint == "" { if endpoint == "" {
endpoint = "http://ollama:11434" endpoint = "http://ollama:11434"
} }
if model == "" { if model == "" {
model = "llama3" model = "llama3"
} }
return &ollamaProvider{ return &OllamaProvider{
endpoint: endpoint, endpoint: endpoint,
model: model, model: model,
client: &http.Client{}, client: &http.Client{},
} }
} }
func (p *ollamaProvider) Name() string { return "ollama" } // NewOllamaManager creates an OllamaProvider for model management (pull/delete/list).
func NewOllamaManager(endpoint string) *OllamaProvider {
return newOllama(endpoint, "")
}
func (p *ollamaProvider) Summarize(ctx context.Context, prompt string) (string, error) {
func (p *OllamaProvider) Name() string { return "ollama" }
func (p *OllamaProvider) Summarize(ctx context.Context, prompt string, opts GenOptions) (string, error) {
numCtx := 32768
if opts.NumCtx > 0 {
numCtx = opts.NumCtx
}
body := map[string]interface{}{ body := map[string]interface{}{
"model": p.model, "model": p.model,
"prompt": prompt, "prompt": prompt,
"stream": false, "stream": false,
"think": opts.Think,
"options": map[string]interface{}{ "options": map[string]interface{}{
"num_ctx": 32768, "num_ctx": numCtx,
}, },
} }
b, _ := json.Marshal(body) b, _ := json.Marshal(body)
@ -67,7 +92,19 @@ func (p *ollamaProvider) Summarize(ctx context.Context, prompt string) (string,
return result.Response, nil return result.Response, nil
} }
func (p *ollamaProvider) ListModels(ctx context.Context) ([]string, error) { func (p *OllamaProvider) ListModels(ctx context.Context) ([]string, error) {
infos, err := p.ListModelsDetailed(ctx)
if err != nil {
return nil, err
}
names := make([]string, len(infos))
for i, m := range infos {
names[i] = m.Name
}
return names, nil
}
func (p *OllamaProvider) ListModelsDetailed(ctx context.Context) ([]OllamaModelInfo, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, p.endpoint+"/api/tags", nil) req, err := http.NewRequestWithContext(ctx, http.MethodGet, p.endpoint+"/api/tags", nil)
if err != nil { if err != nil {
return nil, err return nil, err
@ -80,16 +117,52 @@ func (p *ollamaProvider) ListModels(ctx context.Context) ([]string, error) {
raw, _ := io.ReadAll(resp.Body) raw, _ := io.ReadAll(resp.Body)
var result struct { var result struct {
Models []struct { Models []OllamaModelInfo `json:"models"`
Name string `json:"name"`
} `json:"models"`
} }
if err := json.Unmarshal(raw, &result); err != nil { if err := json.Unmarshal(raw, &result); err != nil {
return nil, err return nil, err
} }
var models []string return result.Models, nil
for _, m := range result.Models { }
models = append(models, m.Name)
} // PullModel pulls (downloads) a model from Ollama Hub. Blocks until complete.
return models, nil func (p *OllamaProvider) PullModel(ctx context.Context, name string) error {
body, _ := json.Marshal(map[string]interface{}{"name": name, "stream": false})
// Use a long-timeout client since model downloads can take many minutes
client := &http.Client{Timeout: 60 * time.Minute}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, p.endpoint+"/api/pull", bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
raw, _ := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("ollama pull error %d: %s", resp.StatusCode, raw)
}
return nil
}
// DeleteModel removes a model from local storage.
func (p *OllamaProvider) DeleteModel(ctx context.Context, name string) error {
body, _ := json.Marshal(map[string]string{"name": name})
req, err := http.NewRequestWithContext(ctx, http.MethodDelete, p.endpoint+"/api/delete", bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
resp, err := p.client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
raw, _ := io.ReadAll(resp.Body)
return fmt.Errorf("ollama delete error %d: %s", resp.StatusCode, raw)
}
return nil
} }

View File

@ -23,7 +23,7 @@ func newOpenAI(apiKey, model string) *openAIProvider {
func (p *openAIProvider) Name() string { return "openai" } func (p *openAIProvider) Name() string { return "openai" }
func (p *openAIProvider) Summarize(ctx context.Context, prompt string) (string, error) { func (p *openAIProvider) Summarize(ctx context.Context, prompt string, _ GenOptions) (string, error) {
resp, err := p.client.CreateChatCompletion(ctx, openai.ChatCompletionRequest{ resp, err := p.client.CreateChatCompletion(ctx, openai.ChatCompletionRequest{
Model: p.model, Model: p.model,
Messages: []openai.ChatCompletionMessage{ Messages: []openai.ChatCompletionMessage{

View File

@ -3,8 +3,10 @@ package ai
import ( import (
"context" "context"
"fmt" "fmt"
"regexp"
"strconv" "strconv"
"strings" "strings"
"sync/atomic"
"time" "time"
"github.com/tradarr/backend/internal/crypto" "github.com/tradarr/backend/internal/crypto"
@ -25,50 +27,53 @@ Structure ton résumé ainsi :
type Pipeline struct { type Pipeline struct {
repo *models.Repository repo *models.Repository
enc *crypto.Encryptor enc *crypto.Encryptor
generating atomic.Bool
} }
func NewPipeline(repo *models.Repository, enc *crypto.Encryptor) *Pipeline { func NewPipeline(repo *models.Repository, enc *crypto.Encryptor) *Pipeline {
return &Pipeline{repo: repo, enc: enc} return &Pipeline{repo: repo, enc: enc}
} }
// BuildProvider instancie un provider à partir de ses paramètres func (p *Pipeline) IsGenerating() bool {
return p.generating.Load()
}
func (p *Pipeline) BuildProvider(name, apiKey, endpoint string) (Provider, error) { func (p *Pipeline) BuildProvider(name, apiKey, endpoint string) (Provider, error) {
provider, err := p.repo.GetActiveAIProvider() return NewProvider(name, apiKey, "", endpoint)
}
// buildProviderForRole resolves and builds the AI provider for a given task role.
func (p *Pipeline) buildProviderForRole(role string) (Provider, *models.AIProvider, error) {
cfg, model, err := p.repo.GetRoleProvider(role)
if err != nil {
return nil, nil, fmt.Errorf("get provider for role %s: %w", role, err)
}
if cfg == nil {
return nil, nil, fmt.Errorf("no AI provider configured for role %s", role)
}
apiKey := ""
if cfg.APIKeyEncrypted != "" {
apiKey, err = p.enc.Decrypt(cfg.APIKeyEncrypted)
if err != nil {
return nil, nil, fmt.Errorf("decrypt API key for role %s: %w", role, err)
}
}
provider, err := NewProvider(cfg.Name, apiKey, model, cfg.Endpoint)
if err != nil {
return nil, nil, fmt.Errorf("build provider for role %s: %w", role, err)
}
return provider, cfg, nil
}
func (p *Pipeline) GenerateForUser(ctx context.Context, userID string) (*models.Summary, error) {
p.generating.Store(true)
defer p.generating.Store(false)
provider, providerCfg, err := p.buildProviderForRole("summary")
if err != nil { if err != nil {
return nil, err return nil, err
} }
model := ""
if provider != nil {
model = provider.Model
}
return NewProvider(name, apiKey, model, endpoint)
}
// GenerateForUser génère un résumé personnalisé pour un utilisateur
func (p *Pipeline) GenerateForUser(ctx context.Context, userID string) (*models.Summary, error) {
// Récupérer le provider actif
providerCfg, err := p.repo.GetActiveAIProvider()
if err != nil {
return nil, fmt.Errorf("get active provider: %w", err)
}
if providerCfg == nil {
return nil, fmt.Errorf("no active AI provider configured")
}
apiKey := ""
if providerCfg.APIKeyEncrypted != "" {
apiKey, err = p.enc.Decrypt(providerCfg.APIKeyEncrypted)
if err != nil {
return nil, fmt.Errorf("decrypt API key: %w", err)
}
}
provider, err := NewProvider(providerCfg.Name, apiKey, providerCfg.Model, providerCfg.Endpoint)
if err != nil {
return nil, fmt.Errorf("build provider: %w", err)
}
// Récupérer la watchlist de l'utilisateur (pour le contexte IA uniquement)
assets, err := p.repo.GetUserAssets(userID) assets, err := p.repo.GetUserAssets(userID)
if err != nil { if err != nil {
return nil, fmt.Errorf("get user assets: %w", err) return nil, fmt.Errorf("get user assets: %w", err)
@ -78,7 +83,6 @@ func (p *Pipeline) GenerateForUser(ctx context.Context, userID string) (*models.
symbols[i] = a.Symbol symbols[i] = a.Symbol
} }
// Récupérer TOUS les articles récents, toutes sources confondues
hoursStr, _ := p.repo.GetSetting("articles_lookback_hours") hoursStr, _ := p.repo.GetSetting("articles_lookback_hours")
hours, _ := strconv.Atoi(hoursStr) hours, _ := strconv.Atoi(hoursStr)
if hours == 0 { if hours == 0 {
@ -98,25 +102,150 @@ func (p *Pipeline) GenerateForUser(ctx context.Context, userID string) (*models.
if maxArticles == 0 { if maxArticles == 0 {
maxArticles = 50 maxArticles = 50
} }
if len(articles) > maxArticles {
// Passe 1 : filtrage par pertinence — seulement si nettement plus d'articles que le max
if len(articles) > maxArticles*2 {
filterProvider, _, filterErr := p.buildProviderForRole("filter")
if filterErr != nil {
filterProvider = provider // fallback to summary provider
}
fmt.Printf("[pipeline] Passe 1 — filtrage : %d articles → sélection des %d plus pertinents…\n", len(articles), maxArticles)
t1 := time.Now()
articles = p.filterByRelevance(ctx, filterProvider, symbols, articles, maxArticles)
fmt.Printf("[pipeline] Passe 1 — terminée en %s : %d articles retenus\n", time.Since(t1).Round(time.Second), len(articles))
} else if len(articles) > maxArticles {
articles = articles[:maxArticles] articles = articles[:maxArticles]
fmt.Printf("[pipeline] troncature directe à %d articles (pas assez d'excédent pour justifier un appel IA)\n", maxArticles)
} }
systemPrompt, _ := p.repo.GetSetting("ai_system_prompt") systemPrompt, _ := p.repo.GetSetting("ai_system_prompt")
if systemPrompt == "" { if systemPrompt == "" {
systemPrompt = DefaultSystemPrompt systemPrompt = DefaultSystemPrompt
} }
prompt := buildPrompt(systemPrompt, symbols, articles)
summary, err := provider.Summarize(ctx, prompt) tz, _ := p.repo.GetSetting("timezone")
// Passe 2 : résumé complet
fmt.Printf("[pipeline] Passe 2 — résumé : génération sur %d articles…\n", len(articles))
t2 := time.Now()
prompt := buildPrompt(systemPrompt, symbols, articles, tz)
// Passe 2 : think activé pour une meilleure qualité d'analyse
summary, err := provider.Summarize(ctx, prompt, GenOptions{Think: true, NumCtx: 32768})
if err != nil { if err != nil {
return nil, fmt.Errorf("AI summarize: %w", err) return nil, fmt.Errorf("AI summarize: %w", err)
} }
fmt.Printf("[pipeline] Passe 2 — terminée en %s\n", time.Since(t2).Round(time.Second))
return p.repo.CreateSummary(userID, summary, &providerCfg.ID) return p.repo.CreateSummary(userID, summary, &providerCfg.ID)
} }
// GenerateForAll génère les résumés pour tous les utilisateurs ayant une watchlist // filterByRelevance splits articles into batches and asks the AI to select relevant
// ones from each batch. Results are pooled then truncated to max.
func (p *Pipeline) filterByRelevance(ctx context.Context, provider Provider, symbols []string, articles []models.Article, max int) []models.Article {
batchSizeStr, _ := p.repo.GetSetting("filter_batch_size")
batchSize, _ := strconv.Atoi(batchSizeStr)
if batchSize <= 0 {
batchSize = 20
}
var selected []models.Article
numBatches := (len(articles) + batchSize - 1) / batchSize
for b := 0; b < numBatches; b++ {
start := b * batchSize
end := start + batchSize
if end > len(articles) {
end = len(articles)
}
batch := articles[start:end]
fmt.Printf("[pipeline] Passe 1 — batch %d/%d (%d articles)…\n", b+1, numBatches, len(batch))
t := time.Now()
chosen := p.filterBatch(ctx, provider, symbols, batch)
fmt.Printf("[pipeline] Passe 1 — batch %d/%d terminé en %s : %d retenus\n", b+1, numBatches, time.Since(t).Round(time.Second), len(chosen))
selected = append(selected, chosen...)
// Stop early if we have plenty of candidates
if len(selected) >= max*2 {
fmt.Printf("[pipeline] Passe 1 — suffisamment de candidats (%d), arrêt anticipé\n", len(selected))
break
}
}
if len(selected) <= max {
return selected
}
return selected[:max]
}
// filterBatch asks the AI to return all relevant articles from a single batch.
func (p *Pipeline) filterBatch(ctx context.Context, provider Provider, symbols []string, batch []models.Article) []models.Article {
prompt := buildFilterBatchPrompt(symbols, batch)
response, err := provider.Summarize(ctx, prompt, GenOptions{Think: false, NumCtx: 4096})
if err != nil {
fmt.Printf("[pipeline] filterBatch — échec (%v), conservation du batch entier\n", err)
return batch
}
indices := parseIndexArray(response, len(batch))
if len(indices) == 0 {
return nil
}
filtered := make([]models.Article, 0, len(indices))
for _, i := range indices {
filtered = append(filtered, batch[i])
}
return filtered
}
func buildFilterBatchPrompt(symbols []string, batch []models.Article) string {
var sb strings.Builder
sb.WriteString("Tu es un assistant de trading financier.\n")
sb.WriteString(fmt.Sprintf("Parmi les %d articles ci-dessous, sélectionne TOUS ceux pertinents pour un trader actif.\n", len(batch)))
if len(symbols) > 0 {
sb.WriteString("Actifs prioritaires : ")
sb.WriteString(strings.Join(symbols, ", "))
sb.WriteString("\n")
}
sb.WriteString("\nRéponds UNIQUEMENT avec un tableau JSON des indices retenus (base 0), exemple : [0, 2, 5]\n")
sb.WriteString("Si aucun article n'est pertinent, réponds : []\n")
sb.WriteString("N'ajoute aucun texte avant ou après le tableau JSON.\n\n")
sb.WriteString("Articles :\n")
for i, a := range batch {
sb.WriteString(fmt.Sprintf("[%d] %s (%s)\n", i, a.Title, a.SourceName))
}
return sb.String()
}
var jsonArrayRe = regexp.MustCompile(`\[[\d\s,]+\]`)
func parseIndexArray(response string, maxIndex int) []int {
match := jsonArrayRe.FindString(response)
if match == "" {
return nil
}
match = strings.Trim(match, "[]")
parts := strings.Split(match, ",")
seen := make(map[int]bool)
var indices []int
for _, p := range parts {
n, err := strconv.Atoi(strings.TrimSpace(p))
if err != nil || n < 0 || n >= maxIndex || seen[n] {
continue
}
seen[n] = true
indices = append(indices, n)
}
return indices
}
func (p *Pipeline) GenerateForAll(ctx context.Context) error { func (p *Pipeline) GenerateForAll(ctx context.Context) error {
users, err := p.repo.ListUsers() users, err := p.repo.ListUsers()
if err != nil { if err != nil {
@ -130,7 +259,43 @@ func (p *Pipeline) GenerateForAll(ctx context.Context) error {
return nil return nil
} }
func buildPrompt(systemPrompt string, symbols []string, articles []models.Article) string { // GenerateReportAsync crée le rapport en DB (status=generating) et lance la génération en arrière-plan.
func (p *Pipeline) GenerateReportAsync(reportID, excerpt, question string, mgr *ReportManager) {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
mgr.Register(reportID, cancel)
go func() {
defer cancel()
defer mgr.Remove(reportID)
answer, err := p.callProviderForReport(ctx, excerpt, question)
if err != nil {
if ctx.Err() != nil {
// annulé volontairement — le rapport est supprimé par le handler
return
}
_ = p.repo.UpdateReport(reportID, "error", "", err.Error())
return
}
_ = p.repo.UpdateReport(reportID, "done", answer, "")
}()
}
func (p *Pipeline) callProviderForReport(ctx context.Context, excerpt, question string) (string, error) {
provider, _, err := p.buildProviderForRole("report")
if err != nil {
return "", err
}
prompt := fmt.Sprintf(
"Tu es un assistant financier expert. L'utilisateur a sélectionné les extraits suivants d'un résumé de marché :\n\n%s\n\nQuestion de l'utilisateur : %s\n\nRéponds en français, de façon précise et orientée trading.",
excerpt, question,
)
return provider.Summarize(ctx, prompt, GenOptions{Think: true, NumCtx: 16384})
}
func buildPrompt(systemPrompt string, symbols []string, articles []models.Article, tz string) string {
var sb strings.Builder var sb strings.Builder
sb.WriteString(systemPrompt) sb.WriteString(systemPrompt)
sb.WriteString("\n\n") sb.WriteString("\n\n")
@ -139,7 +304,11 @@ func buildPrompt(systemPrompt string, symbols []string, articles []models.Articl
sb.WriteString(strings.Join(symbols, ", ")) sb.WriteString(strings.Join(symbols, ", "))
sb.WriteString(".\n\n") sb.WriteString(".\n\n")
} }
sb.WriteString(fmt.Sprintf("Date d'analyse : %s\n\n", time.Now().Format("02/01/2006 15:04"))) loc, err := time.LoadLocation(tz)
if err != nil || tz == "" {
loc = time.UTC
}
sb.WriteString(fmt.Sprintf("Date d'analyse : %s\n\n", time.Now().In(loc).Format("02/01/2006 15:04")))
sb.WriteString("## Actualités\n\n") sb.WriteString("## Actualités\n\n")
for i, a := range articles { for i, a := range articles {

View File

@ -5,9 +5,15 @@ import (
"fmt" "fmt"
) )
// GenOptions permet de contrôler le comportement de génération par appel.
type GenOptions struct {
Think bool // active le mode raisonnement (Qwen3 /think)
NumCtx int // taille du contexte KV (0 = valeur par défaut du provider)
}
type Provider interface { type Provider interface {
Name() string Name() string
Summarize(ctx context.Context, prompt string) (string, error) Summarize(ctx context.Context, prompt string, opts GenOptions) (string, error)
ListModels(ctx context.Context) ([]string, error) ListModels(ctx context.Context) ([]string, error)
} }
@ -21,6 +27,8 @@ func NewProvider(name, apiKey, model, endpoint string) (Provider, error) {
return newGemini(apiKey, model), nil return newGemini(apiKey, model), nil
case "ollama": case "ollama":
return newOllama(endpoint, model), nil return newOllama(endpoint, model), nil
case "claudecode":
return newClaudeCode(model), nil
default: default:
return nil, fmt.Errorf("unknown provider: %s", name) return nil, fmt.Errorf("unknown provider: %s", name)
} }

View File

@ -0,0 +1,37 @@
package ai
import (
"context"
"sync"
)
// ReportManager tracks in-flight report goroutines so they can be cancelled.
type ReportManager struct {
mu sync.Mutex
cancels map[string]context.CancelFunc
}
func NewReportManager() *ReportManager {
return &ReportManager{cancels: make(map[string]context.CancelFunc)}
}
func (m *ReportManager) Register(id string, cancel context.CancelFunc) {
m.mu.Lock()
defer m.mu.Unlock()
m.cancels[id] = cancel
}
func (m *ReportManager) Cancel(id string) {
m.mu.Lock()
defer m.mu.Unlock()
if cancel, ok := m.cancels[id]; ok {
cancel()
delete(m.cancels, id)
}
}
func (m *ReportManager) Remove(id string) {
m.mu.Lock()
defer m.mu.Unlock()
delete(m.cancels, id)
}

View File

@ -1,6 +1,7 @@
package handlers package handlers
import ( import (
"context"
"fmt" "fmt"
"net/http" "net/http"
@ -173,6 +174,25 @@ func (h *Handler) DeleteAIProvider(c *gin.Context) {
httputil.NoContent(c) httputil.NoContent(c)
} }
func (h *Handler) ProbeAIModels(c *gin.Context) {
var req aiProviderRequest
if err := c.ShouldBindJSON(&req); err != nil {
httputil.BadRequest(c, err)
return
}
provider, err := h.pipeline.BuildProvider(req.Name, req.APIKey, req.Endpoint)
if err != nil {
httputil.InternalError(c, err)
return
}
models, err := provider.ListModels(c.Request.Context())
if err != nil {
httputil.InternalError(c, err)
return
}
httputil.OK(c, models)
}
func (h *Handler) ListAIModels(c *gin.Context) { func (h *Handler) ListAIModels(c *gin.Context) {
id := c.Param("id") id := c.Param("id")
p, err := h.repo.GetAIProviderByID(id) p, err := h.repo.GetAIProviderByID(id)
@ -287,10 +307,161 @@ func (h *Handler) UpdateSettings(c *gin.Context) {
httputil.OK(c, gin.H{"ok": true}) httputil.OK(c, gin.H{"ok": true})
} }
// ── Schedule ───────────────────────────────────────────────────────────────
func (h *Handler) GetSchedule(c *gin.Context) {
slots, err := h.repo.ListScheduleSlots()
if err != nil {
httputil.InternalError(c, err)
return
}
httputil.OK(c, slots)
}
type scheduleRequest struct {
Slots []struct {
DayOfWeek int `json:"day_of_week"`
Hour int `json:"hour"`
Minute int `json:"minute"`
} `json:"slots"`
}
func (h *Handler) UpdateSchedule(c *gin.Context) {
var req scheduleRequest
if err := c.ShouldBindJSON(&req); err != nil {
httputil.BadRequest(c, err)
return
}
slots := make([]models.ScheduleSlot, len(req.Slots))
for i, s := range req.Slots {
slots[i] = models.ScheduleSlot{DayOfWeek: s.DayOfWeek, Hour: s.Hour, Minute: s.Minute}
}
if err := h.repo.ReplaceSchedule(slots); err != nil {
httputil.InternalError(c, err)
return
}
if err := h.scheduler.Reload(); err != nil {
fmt.Printf("schedule reload: %v\n", err)
}
httputil.OK(c, gin.H{"ok": true})
}
func (h *Handler) GetDefaultSystemPrompt(c *gin.Context) { func (h *Handler) GetDefaultSystemPrompt(c *gin.Context) {
httputil.OK(c, gin.H{"prompt": ai.DefaultSystemPrompt}) httputil.OK(c, gin.H{"prompt": ai.DefaultSystemPrompt})
} }
// ── AI Roles ───────────────────────────────────────────────────────────────
func (h *Handler) GetAIRoles(c *gin.Context) {
roles := []string{"summary", "report", "filter"}
resp := gin.H{}
for _, role := range roles {
providerID, _ := h.repo.GetSetting("ai_role_" + role + "_provider")
model, _ := h.repo.GetSetting("ai_role_" + role + "_model")
resp[role] = gin.H{"provider_id": providerID, "model": model}
}
httputil.OK(c, resp)
}
type updateAIRoleRequest struct {
ProviderID string `json:"provider_id"`
Model string `json:"model"`
}
func (h *Handler) UpdateAIRole(c *gin.Context) {
role := c.Param("role")
if role != "summary" && role != "report" && role != "filter" {
httputil.BadRequest(c, fmt.Errorf("invalid role: must be summary, report, or filter"))
return
}
var req updateAIRoleRequest
if err := c.ShouldBindJSON(&req); err != nil {
httputil.BadRequest(c, err)
return
}
if err := h.repo.SetRoleProvider(role, req.ProviderID, req.Model); err != nil {
httputil.InternalError(c, err)
return
}
httputil.OK(c, gin.H{"ok": true})
}
// ── Ollama Model Management ────────────────────────────────────────────────
func (h *Handler) getOllamaManager() (*ai.OllamaProvider, error) {
providers, err := h.repo.ListAIProviders()
if err != nil {
return nil, err
}
endpoint := ""
for _, p := range providers {
if p.Name == "ollama" {
endpoint = p.Endpoint
break
}
}
if endpoint == "" {
endpoint = "http://ollama:11434"
}
return ai.NewOllamaManager(endpoint), nil
}
func (h *Handler) ListOllamaModels(c *gin.Context) {
mgr, err := h.getOllamaManager()
if err != nil {
httputil.InternalError(c, err)
return
}
models, err := mgr.ListModelsDetailed(c.Request.Context())
if err != nil {
httputil.InternalError(c, err)
return
}
if models == nil {
models = []ai.OllamaModelInfo{}
}
httputil.OK(c, models)
}
type pullModelRequest struct {
Name string `json:"name" binding:"required"`
}
func (h *Handler) PullOllamaModel(c *gin.Context) {
var req pullModelRequest
if err := c.ShouldBindJSON(&req); err != nil {
httputil.BadRequest(c, err)
return
}
mgr, err := h.getOllamaManager()
if err != nil {
httputil.InternalError(c, err)
return
}
go func() {
if err := mgr.PullModel(context.Background(), req.Name); err != nil {
fmt.Printf("[ollama] pull %s failed: %v\n", req.Name, err)
} else {
fmt.Printf("[ollama] pull %s completed\n", req.Name)
}
}()
c.JSON(202, gin.H{"ok": true, "message": "pull started"})
}
func (h *Handler) DeleteOllamaModel(c *gin.Context) {
name := c.Param("name")
mgr, err := h.getOllamaManager()
if err != nil {
httputil.InternalError(c, err)
return
}
if err := mgr.DeleteModel(c.Request.Context(), name); err != nil {
httputil.InternalError(c, err)
return
}
httputil.NoContent(c)
}
// ── Admin Users ──────────────────────────────────────────────────────────── // ── Admin Users ────────────────────────────────────────────────────────────
func (h *Handler) ListUsers(c *gin.Context) { func (h *Handler) ListUsers(c *gin.Context) {

View File

@ -5,6 +5,7 @@ import (
"github.com/tradarr/backend/internal/config" "github.com/tradarr/backend/internal/config"
"github.com/tradarr/backend/internal/crypto" "github.com/tradarr/backend/internal/crypto"
"github.com/tradarr/backend/internal/models" "github.com/tradarr/backend/internal/models"
"github.com/tradarr/backend/internal/scheduler"
"github.com/tradarr/backend/internal/scraper" "github.com/tradarr/backend/internal/scraper"
) )
@ -14,6 +15,8 @@ type Handler struct {
enc *crypto.Encryptor enc *crypto.Encryptor
registry *scraper.Registry registry *scraper.Registry
pipeline *ai.Pipeline pipeline *ai.Pipeline
scheduler *scheduler.Scheduler
reportManager *ai.ReportManager
} }
func New( func New(
@ -22,6 +25,7 @@ func New(
enc *crypto.Encryptor, enc *crypto.Encryptor,
registry *scraper.Registry, registry *scraper.Registry,
pipeline *ai.Pipeline, pipeline *ai.Pipeline,
sched *scheduler.Scheduler,
) *Handler { ) *Handler {
return &Handler{ return &Handler{
repo: repo, repo: repo,
@ -29,5 +33,7 @@ func New(
enc: enc, enc: enc,
registry: registry, registry: registry,
pipeline: pipeline, pipeline: pipeline,
scheduler: sched,
reportManager: ai.NewReportManager(),
} }
} }

View File

@ -0,0 +1,84 @@
package handlers
import (
"net/http"
"strings"
"github.com/gin-gonic/gin"
"github.com/tradarr/backend/internal/httputil"
)
type reportRequest struct {
SummaryID string `json:"summary_id"`
Excerpts []string `json:"excerpts" binding:"required,min=1"`
Question string `json:"question" binding:"required"`
}
func (h *Handler) CreateReport(c *gin.Context) {
userID := c.GetString("userID")
var req reportRequest
if err := c.ShouldBindJSON(&req); err != nil {
httputil.BadRequest(c, err)
return
}
var summaryID *string
if req.SummaryID != "" {
summaryID = &req.SummaryID
}
// Joindre les extraits avec un séparateur visuel
excerpt := buildExcerptContext(req.Excerpts)
// Créer le rapport en DB avec status=generating, retourner immédiatement
report, err := h.repo.CreatePendingReport(userID, summaryID, excerpt, req.Question)
if err != nil {
httputil.InternalError(c, err)
return
}
// Lancer la génération en arrière-plan
h.pipeline.GenerateReportAsync(report.ID, excerpt, req.Question, h.reportManager)
c.JSON(http.StatusCreated, report)
}
func (h *Handler) ListReports(c *gin.Context) {
userID := c.GetString("userID")
reports, err := h.repo.ListReports(userID)
if err != nil {
httputil.InternalError(c, err)
return
}
httputil.OK(c, reports)
}
func (h *Handler) DeleteReport(c *gin.Context) {
userID := c.GetString("userID")
id := c.Param("id")
// Annuler la goroutine si elle tourne encore
h.reportManager.Cancel(id)
if err := h.repo.DeleteReport(id, userID); err != nil {
httputil.InternalError(c, err)
return
}
c.Status(http.StatusNoContent)
}
func (h *Handler) GetGeneratingStatus(c *gin.Context) {
httputil.OK(c, gin.H{"generating": h.pipeline.IsGenerating()})
}
func buildExcerptContext(excerpts []string) string {
if len(excerpts) == 1 {
return excerpts[0]
}
var sb strings.Builder
for i, e := range excerpts {
if i > 0 {
sb.WriteString("\n\n---\n\n")
}
sb.WriteString(e)
}
return sb.String()
}

View File

@ -7,7 +7,11 @@ import (
) )
func SetupRouter(h *handlers.Handler, jwtSecret string) *gin.Engine { func SetupRouter(h *handlers.Handler, jwtSecret string) *gin.Engine {
r := gin.Default() r := gin.New()
r.Use(gin.Recovery())
r.Use(gin.LoggerWithConfig(gin.LoggerConfig{
SkipPaths: []string{"/api/summaries/status"},
}))
r.Use(func(c *gin.Context) { r.Use(func(c *gin.Context) {
c.Header("Access-Control-Allow-Origin", "*") c.Header("Access-Control-Allow-Origin", "*")
@ -39,8 +43,13 @@ func SetupRouter(h *handlers.Handler, jwtSecret string) *gin.Engine {
authed.GET("/articles/:id", h.GetArticle) authed.GET("/articles/:id", h.GetArticle)
authed.GET("/summaries", h.ListSummaries) authed.GET("/summaries", h.ListSummaries)
authed.GET("/summaries/status", h.GetGeneratingStatus)
authed.POST("/summaries/generate", h.GenerateSummary) authed.POST("/summaries/generate", h.GenerateSummary)
authed.GET("/reports", h.ListReports)
authed.POST("/reports", h.CreateReport)
authed.DELETE("/reports/:id", h.DeleteReport)
// Admin // Admin
admin := authed.Group("/admin") admin := authed.Group("/admin")
admin.Use(auth.AdminOnly()) admin.Use(auth.AdminOnly())
@ -54,6 +63,7 @@ func SetupRouter(h *handlers.Handler, jwtSecret string) *gin.Engine {
admin.POST("/ai-providers/:id/activate", h.SetActiveAIProvider) admin.POST("/ai-providers/:id/activate", h.SetActiveAIProvider)
admin.DELETE("/ai-providers/:id", h.DeleteAIProvider) admin.DELETE("/ai-providers/:id", h.DeleteAIProvider)
admin.GET("/ai-providers/:id/models", h.ListAIModels) admin.GET("/ai-providers/:id/models", h.ListAIModels)
admin.POST("/ai-providers/probe-models", h.ProbeAIModels)
admin.GET("/sources", h.ListSources) admin.GET("/sources", h.ListSources)
admin.PUT("/sources/:id", h.UpdateSource) admin.PUT("/sources/:id", h.UpdateSource)
@ -65,9 +75,21 @@ func SetupRouter(h *handlers.Handler, jwtSecret string) *gin.Engine {
admin.PUT("/settings", h.UpdateSettings) admin.PUT("/settings", h.UpdateSettings)
admin.GET("/settings/default-prompt", h.GetDefaultSystemPrompt) admin.GET("/settings/default-prompt", h.GetDefaultSystemPrompt)
admin.GET("/schedule", h.GetSchedule)
admin.PUT("/schedule", h.UpdateSchedule)
admin.GET("/users", h.ListUsers) admin.GET("/users", h.ListUsers)
admin.PUT("/users/:id", h.UpdateAdminUser) admin.PUT("/users/:id", h.UpdateAdminUser)
admin.DELETE("/users/:id", h.DeleteAdminUser) admin.DELETE("/users/:id", h.DeleteAdminUser)
// AI roles (per-task model assignment)
admin.GET("/ai-roles", h.GetAIRoles)
admin.PUT("/ai-roles/:role", h.UpdateAIRole)
// Ollama model management
admin.GET("/ollama/models", h.ListOllamaModels)
admin.POST("/ollama/pull", h.PullOllamaModel)
admin.DELETE("/ollama/models/:name", h.DeleteOllamaModel)
return r return r
} }

View File

@ -20,7 +20,7 @@ func GenerateToken(userID, email, role, secret string) (string, error) {
Email: email, Email: email,
Role: role, Role: role,
RegisteredClaims: jwt.RegisteredClaims{ RegisteredClaims: jwt.RegisteredClaims{
ExpiresAt: jwt.NewNumericDate(time.Now().Add(24 * time.Hour)), ExpiresAt: jwt.NewNumericDate(time.Now().Add(7 * 24 * time.Hour)),
IssuedAt: jwt.NewNumericDate(time.Now()), IssuedAt: jwt.NewNumericDate(time.Now()),
}, },
} }

View File

@ -11,7 +11,7 @@ type Config struct {
JWTSecret string JWTSecret string
EncryptionKey []byte EncryptionKey []byte
Port string Port string
ChromePath string ScraperURL string
AdminEmail string AdminEmail string
AdminPassword string AdminPassword string
} }
@ -41,12 +41,17 @@ func Load() (*Config, error) {
port = "8080" port = "8080"
} }
scraperURL := os.Getenv("SCRAPER_URL")
if scraperURL == "" {
scraperURL = "http://scraper:3001"
}
return &Config{ return &Config{
DatabaseURL: dbURL, DatabaseURL: dbURL,
JWTSecret: jwtSecret, JWTSecret: jwtSecret,
EncryptionKey: encKey, EncryptionKey: encKey,
Port: port, Port: port,
ChromePath: os.Getenv("CHROME_PATH"), ScraperURL: scraperURL,
AdminEmail: os.Getenv("ADMIN_EMAIL"), AdminEmail: os.Getenv("ADMIN_EMAIL"),
AdminPassword: os.Getenv("ADMIN_PASSWORD"), AdminPassword: os.Getenv("ADMIN_PASSWORD"),
}, nil }, nil

View File

@ -21,7 +21,7 @@ CREATE TABLE user_assets (
CREATE TABLE sources ( CREATE TABLE sources (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(), id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL, name TEXT NOT NULL,
type TEXT NOT NULL CHECK (type IN ('bloomberg', 'stocktwits')), type TEXT NOT NULL CHECK (type IN ('bloomberg', 'stocktwits', 'reuters', 'watcherguru')),
enabled BOOLEAN NOT NULL DEFAULT TRUE, enabled BOOLEAN NOT NULL DEFAULT TRUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
); );
@ -97,7 +97,7 @@ CREATE INDEX idx_user_assets_user_id ON user_assets(user_id);
-- Sources initiales -- Sources initiales
INSERT INTO sources (name, type, enabled) VALUES INSERT INTO sources (name, type, enabled) VALUES
('Bloomberg', 'bloomberg', TRUE), ('Bloomberg', 'bloomberg', TRUE),
('StockTwits', 'stocktwits', TRUE); ('Yahoo Finance', 'stocktwits', TRUE);
-- Paramètres par défaut -- Paramètres par défaut
INSERT INTO settings (key, value) VALUES INSERT INTO settings (key, value) VALUES

View File

@ -0,0 +1 @@
DELETE FROM sources WHERE type IN ('reuters', 'watcherguru');

View File

@ -0,0 +1,4 @@
INSERT INTO sources (name, type, enabled) VALUES
('Reuters', 'reuters', true),
('Watcher.Guru', 'watcherguru', true)
ON CONFLICT DO NOTHING;

View File

@ -0,0 +1 @@
DROP TABLE IF EXISTS scrape_schedules;

View File

@ -0,0 +1,17 @@
CREATE TABLE scrape_schedules (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
day_of_week SMALLINT NOT NULL CHECK (day_of_week BETWEEN 0 AND 6),
hour SMALLINT NOT NULL CHECK (hour BETWEEN 0 AND 23),
minute SMALLINT NOT NULL DEFAULT 0 CHECK (minute BETWEEN 0 AND 59),
UNIQUE (day_of_week, hour, minute)
);
-- Planning par défaut : lun-ven à 6h et 15h, week-end à 6h uniquement
INSERT INTO scrape_schedules (day_of_week, hour, minute) VALUES
(1, 6, 0), (1, 15, 0),
(2, 6, 0), (2, 15, 0),
(3, 6, 0), (3, 15, 0),
(4, 6, 0), (4, 15, 0),
(5, 6, 0), (5, 15, 0),
(6, 6, 0),
(0, 6, 0);

View File

@ -0,0 +1 @@
DROP TABLE IF EXISTS reports;

View File

@ -0,0 +1,9 @@
CREATE TABLE reports (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
summary_id UUID REFERENCES summaries(id) ON DELETE SET NULL,
context_excerpt TEXT NOT NULL,
question TEXT NOT NULL,
answer TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

View File

@ -0,0 +1,4 @@
ALTER TABLE reports
DROP COLUMN IF EXISTS status,
DROP COLUMN IF EXISTS error_msg,
ALTER COLUMN answer DROP DEFAULT;

View File

@ -0,0 +1,4 @@
ALTER TABLE reports
ALTER COLUMN answer SET DEFAULT '',
ADD COLUMN status VARCHAR(20) NOT NULL DEFAULT 'done',
ADD COLUMN error_msg TEXT NOT NULL DEFAULT '';

View File

@ -0,0 +1 @@
DELETE FROM settings WHERE key = 'timezone';

View File

@ -0,0 +1,2 @@
INSERT INTO settings (key, value) VALUES ('timezone', 'Europe/Paris')
ON CONFLICT (key) DO NOTHING;

View File

@ -0,0 +1 @@
DELETE FROM settings WHERE key = 'filter_batch_size';

View File

@ -0,0 +1,2 @@
INSERT INTO settings (key, value) VALUES ('filter_batch_size', '20')
ON CONFLICT (key) DO NOTHING;

View File

@ -0,0 +1,3 @@
ALTER TABLE ai_providers DROP CONSTRAINT IF EXISTS ai_providers_name_check;
ALTER TABLE ai_providers ADD CONSTRAINT ai_providers_name_check
CHECK (name IN ('openai', 'anthropic', 'gemini', 'ollama'));

View File

@ -0,0 +1,3 @@
ALTER TABLE ai_providers DROP CONSTRAINT IF EXISTS ai_providers_name_check;
ALTER TABLE ai_providers ADD CONSTRAINT ai_providers_name_check
CHECK (name IN ('openai', 'anthropic', 'gemini', 'ollama', 'claudecode'));

View File

@ -97,3 +97,22 @@ type Setting struct {
Key string `json:"key"` Key string `json:"key"`
Value string `json:"value"` Value string `json:"value"`
} }
type ScheduleSlot struct {
ID string `json:"id"`
DayOfWeek int `json:"day_of_week"` // 0=dimanche, 1=lundi ... 6=samedi
Hour int `json:"hour"`
Minute int `json:"minute"`
}
type Report struct {
ID string `json:"id"`
UserID string `json:"user_id"`
SummaryID *string `json:"summary_id"`
ContextExcerpt string `json:"context_excerpt"`
Question string `json:"question"`
Answer string `json:"answer"`
Status string `json:"status"` // generating | done | error
ErrorMsg string `json:"error_msg"`
CreatedAt time.Time `json:"created_at"`
}

View File

@ -187,20 +187,28 @@ func (r *Repository) UpdateSource(id string, enabled bool) error {
// ── Articles ─────────────────────────────────────────────────────────────── // ── Articles ───────────────────────────────────────────────────────────────
func (r *Repository) UpsertArticle(sourceID, title, content, url string, publishedAt *time.Time) (*Article, error) { // InsertArticleIfNew insère l'article uniquement s'il n'existe pas déjà (par URL).
a := &Article{} // Retourne (article, true, nil) si inséré, (nil, false, nil) si déjà présent.
func (r *Repository) InsertArticleIfNew(sourceID, title, content, url string, publishedAt *time.Time) (*Article, bool, error) {
var pa sql.NullTime var pa sql.NullTime
if publishedAt != nil { if publishedAt != nil {
pa = sql.NullTime{Time: *publishedAt, Valid: true} pa = sql.NullTime{Time: *publishedAt, Valid: true}
} }
a := &Article{}
err := r.db.QueryRow(` err := r.db.QueryRow(`
INSERT INTO articles (source_id, title, content, url, published_at) INSERT INTO articles (source_id, title, content, url, published_at)
VALUES ($1, $2, $3, $4, $5) VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (url) DO UPDATE SET title=EXCLUDED.title, content=EXCLUDED.content ON CONFLICT (url) DO NOTHING
RETURNING id, source_id, title, content, url, published_at, created_at`, RETURNING id, source_id, title, content, url, published_at, created_at`,
sourceID, title, content, url, pa, sourceID, title, content, url, pa,
).Scan(&a.ID, &a.SourceID, &a.Title, &a.Content, &a.URL, &a.PublishedAt, &a.CreatedAt) ).Scan(&a.ID, &a.SourceID, &a.Title, &a.Content, &a.URL, &a.PublishedAt, &a.CreatedAt)
return a, err if err == sql.ErrNoRows {
return nil, false, nil // déjà présent
}
if err != nil {
return nil, false, err
}
return a, true, nil
} }
func (r *Repository) AddArticleSymbol(articleID, symbol string) error { func (r *Repository) AddArticleSymbol(articleID, symbol string) error {
@ -260,7 +268,7 @@ func (r *Repository) GetRecentArticles(hours int) ([]Article, error) {
SELECT a.id, a.source_id, s.name, a.title, a.content, a.url, a.published_at, a.created_at SELECT a.id, a.source_id, s.name, a.title, a.content, a.url, a.published_at, a.created_at
FROM articles a FROM articles a
JOIN sources s ON s.id = a.source_id JOIN sources s ON s.id = a.source_id
WHERE a.created_at > NOW() - ($1 * INTERVAL '1 hour') WHERE COALESCE(a.published_at, a.created_at) > NOW() - ($1 * INTERVAL '1 hour')
ORDER BY a.published_at DESC NULLS LAST, a.created_at DESC`, hours) ORDER BY a.published_at DESC NULLS LAST, a.created_at DESC`, hours)
if err != nil { if err != nil {
return nil, err return nil, err
@ -520,6 +528,51 @@ func (r *Repository) SetSetting(key, value string) error {
return err return err
} }
// ── Schedule ───────────────────────────────────────────────────────────────
func (r *Repository) ListScheduleSlots() ([]ScheduleSlot, error) {
rows, err := r.db.Query(`
SELECT id, day_of_week, hour, minute FROM scrape_schedules
ORDER BY day_of_week, hour, minute`)
if err != nil {
return nil, err
}
defer rows.Close()
var slots []ScheduleSlot
for rows.Next() {
var s ScheduleSlot
if err := rows.Scan(&s.ID, &s.DayOfWeek, &s.Hour, &s.Minute); err != nil {
return nil, err
}
slots = append(slots, s)
}
return slots, nil
}
func (r *Repository) ReplaceSchedule(slots []ScheduleSlot) error {
tx, err := r.db.Begin()
if err != nil {
return err
}
defer tx.Rollback()
if _, err := tx.Exec(`DELETE FROM scrape_schedules`); err != nil {
return err
}
for _, s := range slots {
if _, err := tx.Exec(
`INSERT INTO scrape_schedules (day_of_week, hour, minute) VALUES ($1, $2, $3)
ON CONFLICT (day_of_week, hour, minute) DO NOTHING`,
s.DayOfWeek, s.Hour, s.Minute,
); err != nil {
return err
}
}
return tx.Commit()
}
// ── Settings ───────────────────────────────────────────────────────────────
func (r *Repository) ListSettings() ([]Setting, error) { func (r *Repository) ListSettings() ([]Setting, error) {
rows, err := r.db.Query(`SELECT key, value FROM settings ORDER BY key`) rows, err := r.db.Query(`SELECT key, value FROM settings ORDER BY key`)
if err != nil { if err != nil {
@ -536,3 +589,80 @@ func (r *Repository) ListSettings() ([]Setting, error) {
} }
return settings, nil return settings, nil
} }
// ── AI Role Providers ──────────────────────────────────────────────────────
// GetRoleProvider returns the configured provider and model for a given role (summary/report/filter).
// Falls back to the active provider if no role-specific provider is set.
func (r *Repository) GetRoleProvider(role string) (*AIProvider, string, error) {
providerID, _ := r.GetSetting("ai_role_" + role + "_provider")
model, _ := r.GetSetting("ai_role_" + role + "_model")
if providerID != "" {
p, err := r.GetAIProviderByID(providerID)
if err == nil && p != nil {
if model == "" {
model = p.Model
}
return p, model, nil
}
}
// Fallback to active provider
p, err := r.GetActiveAIProvider()
if p != nil && model == "" {
model = p.Model
}
return p, model, err
}
func (r *Repository) SetRoleProvider(role, providerID, model string) error {
if err := r.SetSetting("ai_role_"+role+"_provider", providerID); err != nil {
return err
}
return r.SetSetting("ai_role_"+role+"_model", model)
}
// ── Reports ────────────────────────────────────────────────────────────────
func (r *Repository) CreatePendingReport(userID string, summaryID *string, excerpt, question string) (*Report, error) {
rep := &Report{}
err := r.db.QueryRow(`
INSERT INTO reports (user_id, summary_id, context_excerpt, question, answer, status)
VALUES ($1, $2, $3, $4, '', 'generating')
RETURNING id, user_id, summary_id, context_excerpt, question, answer, status, error_msg, created_at`,
userID, summaryID, excerpt, question,
).Scan(&rep.ID, &rep.UserID, &rep.SummaryID, &rep.ContextExcerpt, &rep.Question, &rep.Answer, &rep.Status, &rep.ErrorMsg, &rep.CreatedAt)
return rep, err
}
func (r *Repository) UpdateReport(id, status, answer, errorMsg string) error {
_, err := r.db.Exec(`
UPDATE reports SET status=$1, answer=$2, error_msg=$3 WHERE id=$4`,
status, answer, errorMsg, id)
return err
}
func (r *Repository) ListReports(userID string) ([]Report, error) {
rows, err := r.db.Query(`
SELECT id, user_id, summary_id, context_excerpt, question, answer, status, error_msg, created_at
FROM reports WHERE user_id=$1
ORDER BY created_at DESC`, userID)
if err != nil {
return nil, err
}
defer rows.Close()
var reports []Report
for rows.Next() {
var rep Report
if err := rows.Scan(&rep.ID, &rep.UserID, &rep.SummaryID, &rep.ContextExcerpt, &rep.Question, &rep.Answer, &rep.Status, &rep.ErrorMsg, &rep.CreatedAt); err != nil {
return nil, err
}
reports = append(reports, rep)
}
return reports, nil
}
func (r *Repository) DeleteReport(id, userID string) error {
_, err := r.db.Exec(`DELETE FROM reports WHERE id=$1 AND user_id=$2`, id, userID)
return err
}

View File

@ -3,7 +3,6 @@ package scheduler
import ( import (
"context" "context"
"fmt" "fmt"
"strconv"
"github.com/robfig/cron/v3" "github.com/robfig/cron/v3"
"github.com/tradarr/backend/internal/ai" "github.com/tradarr/backend/internal/ai"
@ -16,7 +15,7 @@ type Scheduler struct {
registry *scraper.Registry registry *scraper.Registry
pipeline *ai.Pipeline pipeline *ai.Pipeline
repo *models.Repository repo *models.Repository
entryID cron.EntryID entryIDs []cron.EntryID
} }
func New(registry *scraper.Registry, pipeline *ai.Pipeline, repo *models.Repository) *Scheduler { func New(registry *scraper.Registry, pipeline *ai.Pipeline, repo *models.Repository) *Scheduler {
@ -29,19 +28,10 @@ func New(registry *scraper.Registry, pipeline *ai.Pipeline, repo *models.Reposit
} }
func (s *Scheduler) Start() error { func (s *Scheduler) Start() error {
interval, err := s.getInterval() if err := s.loadSchedule(); err != nil {
if err != nil {
return err return err
} }
spec := fmt.Sprintf("@every %dm", interval)
s.entryID, err = s.cron.AddFunc(spec, s.run)
if err != nil {
return fmt.Errorf("add cron: %w", err)
}
s.cron.Start() s.cron.Start()
fmt.Printf("scheduler started, running every %d minutes\n", interval)
return nil return nil
} }
@ -50,39 +40,51 @@ func (s *Scheduler) Stop() {
} }
func (s *Scheduler) Reload() error { func (s *Scheduler) Reload() error {
s.cron.Remove(s.entryID) for _, id := range s.entryIDs {
interval, err := s.getInterval() s.cron.Remove(id)
if err != nil {
return err
} }
spec := fmt.Sprintf("@every %dm", interval) s.entryIDs = nil
s.entryID, err = s.cron.AddFunc(spec, s.run) return s.loadSchedule()
return err }
func (s *Scheduler) loadSchedule() error {
slots, err := s.repo.ListScheduleSlots()
if err != nil {
return fmt.Errorf("load schedule: %w", err)
}
if len(slots) == 0 {
fmt.Println("scheduler: no schedule configured, scraping disabled")
return nil
}
tz, _ := s.repo.GetSetting("timezone")
if tz == "" {
tz = "UTC"
}
for _, slot := range slots {
// TZ= prefix permet à robfig/cron d'interpréter les heures dans le fuseau configuré
spec := fmt.Sprintf("TZ=%s %d %d * * %d", tz, slot.Minute, slot.Hour, slot.DayOfWeek)
id, err := s.cron.AddFunc(spec, s.run)
if err != nil {
fmt.Printf("scheduler: invalid cron spec %q: %v\n", spec, err)
continue
}
s.entryIDs = append(s.entryIDs, id)
}
fmt.Printf("scheduler: %d time slots loaded (timezone: %s)\n", len(s.entryIDs), tz)
return nil
} }
func (s *Scheduler) run() { func (s *Scheduler) run() {
fmt.Println("scheduler: running scraping cycle") fmt.Println("scheduler: starting scraping cycle")
if err := s.registry.RunAll(); err != nil { if err := s.registry.RunAll(); err != nil {
fmt.Printf("scheduler scrape error: %v\n", err) fmt.Printf("scheduler scrape error: %v\n", err)
return return
} }
fmt.Println("scheduler: running AI summaries") fmt.Println("scheduler: starting AI summaries")
if err := s.pipeline.GenerateForAll(context.Background()); err != nil { if err := s.pipeline.GenerateForAll(context.Background()); err != nil {
fmt.Printf("scheduler summary error: %v\n", err) fmt.Printf("scheduler summary error: %v\n", err)
} }
} }
func (s *Scheduler) getInterval() (int, error) {
v, err := s.repo.GetSetting("scrape_interval_minutes")
if err != nil {
return 60, nil
}
if v == "" {
return 60, nil
}
n, err := strconv.Atoi(v)
if err != nil || n < 1 {
return 60, nil
}
return n, nil
}

View File

@ -1,206 +1,94 @@
package bloomberg package bloomberg
import ( import (
"bytes"
"context" "context"
"encoding/json"
"fmt" "fmt"
"io"
"net/http"
"strings" "strings"
"time" "time"
"github.com/chromedp/chromedp"
"github.com/tradarr/backend/internal/scraper" "github.com/tradarr/backend/internal/scraper"
) )
type Bloomberg struct { type Bloomberg struct {
username string scraperURL string
password string client *http.Client
chromePath string
} }
func New(username, password, chromePath string) *Bloomberg { func New(scraperURL string) *Bloomberg {
return &Bloomberg{username: username, password: password, chromePath: chromePath} if scraperURL == "" {
scraperURL = "http://scraper:3001"
}
return &Bloomberg{
scraperURL: scraperURL,
client: &http.Client{Timeout: 10 * time.Minute},
}
} }
func (b *Bloomberg) Name() string { return "bloomberg" } func (b *Bloomberg) Name() string { return "bloomberg" }
func (b *Bloomberg) Scrape(ctx context.Context, symbols []string) ([]scraper.Article, error) { type scraperRequest struct {
if b.username == "" || b.password == "" { Username string `json:"username"`
return nil, fmt.Errorf("bloomberg credentials not configured") Password string `json:"password"`
}
opts := []chromedp.ExecAllocatorOption{
chromedp.NoFirstRun,
chromedp.NoDefaultBrowserCheck,
chromedp.Headless,
chromedp.DisableGPU,
chromedp.Flag("no-sandbox", true),
chromedp.Flag("disable-setuid-sandbox", true),
chromedp.Flag("disable-dev-shm-usage", true),
chromedp.Flag("disable-blink-features", "AutomationControlled"),
chromedp.Flag("disable-infobars", true),
chromedp.Flag("window-size", "1920,1080"),
chromedp.Flag("ignore-certificate-errors", true),
chromedp.UserAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"),
}
if b.chromePath != "" {
opts = append(opts, chromedp.ExecPath(b.chromePath))
}
allocCtx, cancelAlloc := chromedp.NewExecAllocator(ctx, opts...)
defer cancelAlloc()
chromeCtx, cancelChrome := chromedp.NewContext(allocCtx)
defer cancelChrome()
timeoutCtx, cancelTimeout := context.WithTimeout(chromeCtx, 5*time.Minute)
defer cancelTimeout()
if err := b.login(timeoutCtx); err != nil {
return nil, fmt.Errorf("bloomberg login: %w", err)
}
var articles []scraper.Article
pages := []string{
"https://www.bloomberg.com/markets",
"https://www.bloomberg.com/technology",
"https://www.bloomberg.com/economics",
}
for _, u := range pages {
pageArticles, err := b.scrapePage(timeoutCtx, u, symbols)
if err != nil {
fmt.Printf("bloomberg scrape %s: %v\n", u, err)
continue
}
articles = append(articles, pageArticles...)
}
fmt.Printf("bloomberg: %d articles fetched total\n", len(articles))
return articles, nil
} }
func (b *Bloomberg) login(ctx context.Context) error { type scraperArticle struct {
loginCtx, cancel := context.WithTimeout(ctx, 2*time.Minute) Title string `json:"title"`
defer cancel() URL string `json:"url"`
// Masquer la détection d'automation via JS
if err := chromedp.Run(loginCtx,
chromedp.ActionFunc(func(ctx context.Context) error {
return chromedp.Evaluate(`
Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
window.chrome = { runtime: {} };
`, nil).Do(ctx)
}),
); err != nil {
fmt.Printf("bloomberg: could not inject stealth JS: %v\n", err)
}
err := chromedp.Run(loginCtx,
chromedp.Navigate("https://www.bloomberg.com/account/signin"),
chromedp.Sleep(2*time.Second),
// Essayer plusieurs sélecteurs pour l'email
chromedp.ActionFunc(func(ctx context.Context) error {
selectors := []string{
`input[name="email"]`,
`input[type="email"]`,
`input[data-type="email"]`,
`input[placeholder*="email" i]`,
`input[placeholder*="mail" i]`,
}
for _, sel := range selectors {
var count int
if err := chromedp.Evaluate(fmt.Sprintf(`document.querySelectorAll('%s').length`, sel), &count).Do(ctx); err == nil && count > 0 {
fmt.Printf("bloomberg: using email selector: %s\n", sel)
return chromedp.SendKeys(sel, b.username, chromedp.ByQuery).Do(ctx)
}
}
return fmt.Errorf("could not find email input — Bloomberg login page structure may have changed")
}),
chromedp.Sleep(500*time.Millisecond),
// Submit email
chromedp.ActionFunc(func(ctx context.Context) error {
selectors := []string{`button[type="submit"]`, `input[type="submit"]`, `button[data-testid*="submit"]`}
for _, sel := range selectors {
var count int
if err := chromedp.Evaluate(fmt.Sprintf(`document.querySelectorAll('%s').length`, sel), &count).Do(ctx); err == nil && count > 0 {
return chromedp.Click(sel, chromedp.ByQuery).Do(ctx)
}
}
// Fallback: press Enter
return chromedp.KeyEvent("\r").Do(ctx)
}),
chromedp.Sleep(2*time.Second),
// Password
chromedp.ActionFunc(func(ctx context.Context) error {
selectors := []string{`input[type="password"]`, `input[name="password"]`}
for _, sel := range selectors {
var count int
if err := chromedp.Evaluate(fmt.Sprintf(`document.querySelectorAll('%s').length`, sel), &count).Do(ctx); err == nil && count > 0 {
fmt.Printf("bloomberg: using password selector: %s\n", sel)
return chromedp.SendKeys(sel, b.password, chromedp.ByQuery).Do(ctx)
}
}
return fmt.Errorf("could not find password input")
}),
chromedp.Sleep(500*time.Millisecond),
chromedp.ActionFunc(func(ctx context.Context) error {
selectors := []string{`button[type="submit"]`, `input[type="submit"]`}
for _, sel := range selectors {
var count int
if err := chromedp.Evaluate(fmt.Sprintf(`document.querySelectorAll('%s').length`, sel), &count).Do(ctx); err == nil && count > 0 {
return chromedp.Click(sel, chromedp.ByQuery).Do(ctx)
}
}
return chromedp.KeyEvent("\r").Do(ctx)
}),
chromedp.Sleep(3*time.Second),
)
return err
} }
func (b *Bloomberg) scrapePage(ctx context.Context, pageURL string, symbols []string) ([]scraper.Article, error) { type scraperResponse struct {
pageCtx, cancel := context.WithTimeout(ctx, 60*time.Second) Articles []scraperArticle `json:"articles"`
defer cancel() Error string `json:"error,omitempty"`
}
var articleNodes []map[string]string func (b *Bloomberg) ScrapeWithCredentials(ctx context.Context, username, password string, symbols []string) ([]scraper.Article, error) {
err := chromedp.Run(pageCtx, payload, _ := json.Marshal(scraperRequest{Username: username, Password: password})
chromedp.Navigate(pageURL), req, err := http.NewRequestWithContext(ctx, http.MethodPost, b.scraperURL+"/bloomberg/scrape", bytes.NewReader(payload))
chromedp.Sleep(3*time.Second),
chromedp.Evaluate(`
(function() {
var items = [];
var seen = new Set();
var links = document.querySelectorAll('a[href*="/news/articles"], a[href*="/opinion/"], a[href*="/markets/"]');
links.forEach(function(a) {
if (seen.has(a.href)) return;
seen.add(a.href);
var title = a.querySelector('h1,h2,h3,h4,[class*="headline"],[class*="title"]');
var text = title ? title.innerText.trim() : a.innerText.trim();
if (text.length > 20 && a.href.includes('bloomberg.com')) {
items.push({title: text, url: a.href});
}
});
return items.slice(0, 25);
})()
`, &articleNodes),
)
if err != nil { if err != nil {
return nil, fmt.Errorf("navigate %s: %w", pageURL, err) return nil, err
}
req.Header.Set("Content-Type", "application/json")
resp, err := b.client.Do(req)
if err != nil {
return nil, fmt.Errorf("scraper service unreachable: %w", err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("scraper service HTTP %d: %s", resp.StatusCode, body)
}
var result scraperResponse
if err := json.Unmarshal(body, &result); err != nil {
return nil, fmt.Errorf("parse scraper response: %w", err)
}
if result.Error != "" {
return nil, fmt.Errorf("bloomberg: %s", result.Error)
} }
var articles []scraper.Article
now := time.Now() now := time.Now()
for _, node := range articleNodes { var articles []scraper.Article
title := strings.TrimSpace(node["title"]) for _, a := range result.Articles {
url := node["url"] title := strings.TrimSpace(a.Title)
if title == "" || url == "" || !strings.Contains(url, "bloomberg.com") { url := a.URL
if title == "" || url == "" {
continue continue
} }
syms := scraper.DetectSymbols(title, symbols) syms := scraper.DetectSymbols(title, symbols)
articles = append(articles, scraper.Article{ articles = append(articles, scraper.Article{
Title: title, Title: title,
Content: title, // contenu minimal — l'article complet nécessite un accès payant Content: title,
URL: url, URL: url,
PublishedAt: &now, PublishedAt: &now,
Symbols: syms, Symbols: syms,
}) })
} }
fmt.Printf("bloomberg: %d articles fetched\n", len(articles))
return articles, nil return articles, nil
} }

View File

@ -9,21 +9,19 @@ import (
"github.com/tradarr/backend/internal/scraper" "github.com/tradarr/backend/internal/scraper"
) )
// DynamicBloomberg charge les credentials depuis la DB avant chaque scraping
type DynamicBloomberg struct { type DynamicBloomberg struct {
repo *models.Repository repo *models.Repository
enc *crypto.Encryptor enc *crypto.Encryptor
chromePath string scraperURL string
} }
func NewDynamic(repo *models.Repository, enc *crypto.Encryptor, chromePath string) *DynamicBloomberg { func NewDynamic(repo *models.Repository, enc *crypto.Encryptor, scraperURL string) *DynamicBloomberg {
return &DynamicBloomberg{repo: repo, enc: enc, chromePath: chromePath} return &DynamicBloomberg{repo: repo, enc: enc, scraperURL: scraperURL}
} }
func (d *DynamicBloomberg) Name() string { return "bloomberg" } func (d *DynamicBloomberg) Name() string { return "bloomberg" }
func (d *DynamicBloomberg) Scrape(ctx context.Context, symbols []string) ([]scraper.Article, error) { func (d *DynamicBloomberg) Scrape(ctx context.Context, symbols []string) ([]scraper.Article, error) {
// Récupérer la source Bloomberg
source, err := d.repo.GetSourceByType("bloomberg") source, err := d.repo.GetSourceByType("bloomberg")
if err != nil || source == nil { if err != nil || source == nil {
return nil, fmt.Errorf("bloomberg source not found") return nil, fmt.Errorf("bloomberg source not found")
@ -34,7 +32,7 @@ func (d *DynamicBloomberg) Scrape(ctx context.Context, symbols []string) ([]scra
return nil, fmt.Errorf("get bloomberg credentials: %w", err) return nil, fmt.Errorf("get bloomberg credentials: %w", err)
} }
if cred == nil || cred.Username == "" { if cred == nil || cred.Username == "" {
return nil, fmt.Errorf("bloomberg credentials not configured — please set them in the admin panel") return nil, fmt.Errorf("bloomberg credentials not configured — configure them in the admin panel")
} }
password := "" password := ""
@ -45,6 +43,6 @@ func (d *DynamicBloomberg) Scrape(ctx context.Context, symbols []string) ([]scra
} }
} }
b := New(cred.Username, password, d.chromePath) b := New(d.scraperURL)
return b.Scrape(ctx, symbols) return b.ScrapeWithCredentials(ctx, cred.Username, password, symbols)
} }

View File

@ -72,11 +72,11 @@ func (r *Registry) Run(sourceID string) error {
return scrapeErr return scrapeErr
} }
// Persister les articles // Persister uniquement les nouveaux articles
count := 0 count := 0
for _, a := range articles { for _, a := range articles {
saved, err := r.repo.UpsertArticle(sourceID, a.Title, a.Content, a.URL, a.PublishedAt) saved, isNew, err := r.repo.InsertArticleIfNew(sourceID, a.Title, a.Content, a.URL, a.PublishedAt)
if err != nil { if err != nil || !isNew {
continue continue
} }
count++ count++

View File

@ -0,0 +1,129 @@
package reuters
import (
"context"
"encoding/xml"
"fmt"
"io"
"net/http"
"strings"
"time"
"github.com/tradarr/backend/internal/scraper"
)
// Reuters RSS est bloqué par Cloudflare. On utilise des flux RSS financiers
// publics fiables à la place : MarketWatch, CNBC, Seeking Alpha.
var feeds = []struct {
name string
url string
}{
{"MarketWatch Top Stories", "https://feeds.content.dowjones.io/public/rss/mw_topstories"},
{"MarketWatch Markets", "https://feeds.content.dowjones.io/public/rss/mw_marketpulse"},
{"CNBC Top News", "https://search.cnbc.com/rs/search/combinedcombined/rss/topNews"},
{"CNBC Finance", "https://search.cnbc.com/rs/search/combinedcombined/rss/topNews?tag=Finance"},
}
type Reuters struct {
client *http.Client
}
func New() *Reuters {
return &Reuters{client: &http.Client{Timeout: 15 * time.Second}}
}
func (r *Reuters) Name() string { return "reuters" }
type rssFeed struct {
Channel struct {
Items []struct {
Title string `xml:"title"`
Link string `xml:"link"`
Description string `xml:"description"`
PubDate string `xml:"pubDate"`
} `xml:"item"`
} `xml:"channel"`
}
func (r *Reuters) Scrape(ctx context.Context, _ []string) ([]scraper.Article, error) {
var articles []scraper.Article
seen := make(map[string]bool)
for i, feed := range feeds {
if i > 0 {
select {
case <-ctx.Done():
return articles, ctx.Err()
case <-time.After(300 * time.Millisecond):
}
}
items, err := r.fetchFeed(ctx, feed.url)
if err != nil {
fmt.Printf("reuters/financial %s: %v\n", feed.name, err)
continue
}
for _, a := range items {
if !seen[a.URL] {
seen[a.URL] = true
articles = append(articles, a)
}
}
fmt.Printf("reuters/financial %s: %d articles\n", feed.name, len(items))
}
return articles, nil
}
func (r *Reuters) fetchFeed(ctx context.Context, feedURL string) ([]scraper.Article, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, feedURL, nil)
if err != nil {
return nil, err
}
req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; Tradarr/1.0)")
req.Header.Set("Accept", "application/rss+xml, application/xml, text/xml")
resp, err := r.client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(io.LimitReader(resp.Body, 256))
return nil, fmt.Errorf("HTTP %d: %s", resp.StatusCode, body)
}
var feed rssFeed
if err := xml.NewDecoder(resp.Body).Decode(&feed); err != nil {
return nil, fmt.Errorf("parse RSS: %w", err)
}
var articles []scraper.Article
for _, item := range feed.Channel.Items {
title := strings.TrimSpace(item.Title)
link := strings.TrimSpace(item.Link)
if title == "" || link == "" {
continue
}
var publishedAt *time.Time
for _, f := range []string{time.RFC1123Z, time.RFC1123, "Mon, 02 Jan 2006 15:04:05 -0700"} {
if t, err := time.Parse(f, item.PubDate); err == nil {
publishedAt = &t
break
}
}
content := strings.TrimSpace(item.Description)
if content == "" {
content = title
}
articles = append(articles, scraper.Article{
Title: title,
Content: content,
URL: link,
PublishedAt: publishedAt,
})
}
return articles, nil
}

View File

@ -0,0 +1,200 @@
package watcherguru
import (
"context"
"encoding/xml"
"fmt"
"io"
"net/http"
"strings"
"time"
"golang.org/x/net/html"
"github.com/tradarr/backend/internal/scraper"
)
const baseURL = "https://watcher.guru"
type WatcherGuru struct {
client *http.Client
}
func New() *WatcherGuru {
return &WatcherGuru{client: &http.Client{Timeout: 15 * time.Second}}
}
func (w *WatcherGuru) Name() string { return "watcherguru" }
type rssFeed struct {
Channel struct {
Items []struct {
Title string `xml:"title"`
Link string `xml:"link"`
PubDate string `xml:"pubDate"`
Desc string `xml:"description"`
} `xml:"item"`
} `xml:"channel"`
}
func (w *WatcherGuru) Scrape(ctx context.Context, _ []string) ([]scraper.Article, error) {
// Try RSS feeds first
for _, feedURL := range []string{
baseURL + "/feed/",
baseURL + "/news/feed/",
} {
articles, err := w.fetchRSS(ctx, feedURL)
if err == nil && len(articles) > 0 {
fmt.Printf("watcherguru rss: %d articles\n", len(articles))
return articles, nil
}
}
// Fallback: HTML scraping
articles, err := w.scrapeHTML(ctx)
if err != nil {
return nil, fmt.Errorf("watcherguru: %w", err)
}
fmt.Printf("watcherguru html: %d articles\n", len(articles))
return articles, nil
}
func (w *WatcherGuru) fetchRSS(ctx context.Context, feedURL string) ([]scraper.Article, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, feedURL, nil)
if err != nil {
return nil, err
}
req.Header.Set("User-Agent", "Mozilla/5.0 (compatible; Tradarr/1.0)")
resp, err := w.client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("HTTP %d", resp.StatusCode)
}
var feed rssFeed
if err := xml.NewDecoder(resp.Body).Decode(&feed); err != nil {
return nil, fmt.Errorf("parse RSS: %w", err)
}
var articles []scraper.Article
for _, item := range feed.Channel.Items {
title := strings.TrimSpace(item.Title)
link := strings.TrimSpace(item.Link)
if title == "" || link == "" {
continue
}
var publishedAt *time.Time
for _, f := range []string{time.RFC1123Z, time.RFC1123, "Mon, 02 Jan 2006 15:04:05 -0700"} {
if t, err := time.Parse(f, item.PubDate); err == nil {
publishedAt = &t
break
}
}
content := strings.TrimSpace(item.Desc)
if content == "" {
content = title
}
articles = append(articles, scraper.Article{
Title: title,
Content: content,
URL: link,
PublishedAt: publishedAt,
})
}
return articles, nil
}
func (w *WatcherGuru) scrapeHTML(ctx context.Context) ([]scraper.Article, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, baseURL+"/news/", nil)
if err != nil {
return nil, err
}
req.Header.Set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/122.0.0.0 Safari/537.36")
req.Header.Set("Accept", "text/html,application/xhtml+xml")
resp, err := w.client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
return nil, fmt.Errorf("HTTP %d: %s", resp.StatusCode, body)
}
doc, err := html.Parse(resp.Body)
if err != nil {
return nil, fmt.Errorf("parse HTML: %w", err)
}
var articles []scraper.Article
seen := make(map[string]bool)
now := time.Now()
var walk func(*html.Node)
walk = func(n *html.Node) {
if n.Type == html.ElementNode && (n.Data == "a" || n.Data == "article") {
if n.Data == "a" {
href := attrVal(n, "href")
if href == "" || seen[href] {
walk(n.FirstChild)
return
}
// Collect links that look like news articles
if strings.Contains(href, "/news/") || strings.Contains(href, "watcher.guru") {
text := strings.TrimSpace(nodeText(n))
if len(text) > 20 {
url := href
if !strings.HasPrefix(url, "http") {
url = baseURL + url
}
if !seen[url] {
seen[url] = true
articles = append(articles, scraper.Article{
Title: text,
Content: text,
URL: url,
PublishedAt: &now,
})
}
}
}
}
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
walk(c)
}
}
walk(doc)
if len(articles) > 40 {
articles = articles[:40]
}
return articles, nil
}
func attrVal(n *html.Node, key string) string {
for _, a := range n.Attr {
if a.Key == key {
return a.Val
}
}
return ""
}
func nodeText(n *html.Node) string {
if n.Type == html.TextNode {
return n.Data
}
var sb strings.Builder
for c := n.FirstChild; c != nil; c = c.NextSibling {
sb.WriteString(nodeText(c))
}
return sb.String()
}

View File

@ -86,8 +86,13 @@ func (y *YahooFinance) fetchSymbol(ctx context.Context, symbol string) ([]scrape
return nil, fmt.Errorf("parse RSS: %w", err) return nil, fmt.Errorf("parse RSS: %w", err)
} }
const maxPerSymbol = 5
var articles []scraper.Article var articles []scraper.Article
for _, item := range feed.Channel.Items { for _, item := range feed.Channel.Items {
if len(articles) >= maxPerSymbol {
break
}
title := strings.TrimSpace(item.Title) title := strings.TrimSpace(item.Title)
link := strings.TrimSpace(item.Link) link := strings.TrimSpace(item.Link)
if title == "" || link == "" { if title == "" || link == "" {

60
docker-compose.prod.yml Normal file
View File

@ -0,0 +1,60 @@
services:
postgres:
image: postgres:16-alpine
restart: unless-stopped
environment:
POSTGRES_DB: ${POSTGRES_DB:-tradarr}
POSTGRES_USER: ${POSTGRES_USER:-tradarr}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is required}
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-tradarr}"]
interval: 10s
timeout: 5s
retries: 5
scraper:
image: gitea.anthonybouteiller.ovh/blomios/tradarr-scraper:v1.0.0
restart: unless-stopped
expose:
- "3001"
backend:
image: gitea.anthonybouteiller.ovh/blomios/tradarr-backend:v1.0.0
restart: unless-stopped
depends_on:
postgres:
condition: service_healthy
scraper:
condition: service_started
environment:
DATABASE_URL: "host=postgres port=5432 user=${POSTGRES_USER:-tradarr} password=${POSTGRES_PASSWORD} dbname=${POSTGRES_DB:-tradarr} sslmode=disable"
JWT_SECRET: ${JWT_SECRET:?JWT_SECRET is required}
ENCRYPTION_KEY: ${ENCRYPTION_KEY:?ENCRYPTION_KEY must be 32 bytes hex}
PORT: "8080"
SCRAPER_URL: "http://scraper:3001"
ADMIN_EMAIL: ${ADMIN_EMAIL:-admin@tradarr.local}
ADMIN_PASSWORD: ${ADMIN_PASSWORD:-changeme}
volumes:
- /home/anthony/.claude:/root/.claude
expose:
- "8080"
ollama:
image: ollama/ollama
restart: unless-stopped
volumes:
- ollama_data:/root/.ollama
frontend:
image: gitea.anthonybouteiller.ovh/blomios/tradarr-frontend:v1.0.0
restart: unless-stopped
depends_on:
- backend
ports:
- "${FRONTEND_PORT:-80}:80"
volumes:
pgdata:
ollama_data:

View File

@ -14,6 +14,14 @@ services:
timeout: 5s timeout: 5s
retries: 5 retries: 5
scraper:
build:
context: ./scraper-service
dockerfile: Dockerfile
restart: unless-stopped
expose:
- "3001"
backend: backend:
build: build:
context: ./backend context: ./backend
@ -22,13 +30,18 @@ services:
depends_on: depends_on:
postgres: postgres:
condition: service_healthy condition: service_healthy
scraper:
condition: service_started
environment: environment:
DATABASE_URL: "host=postgres port=5432 user=${POSTGRES_USER:-tradarr} password=${POSTGRES_PASSWORD} dbname=${POSTGRES_DB:-tradarr} sslmode=disable" DATABASE_URL: "host=postgres port=5432 user=${POSTGRES_USER:-tradarr} password=${POSTGRES_PASSWORD} dbname=${POSTGRES_DB:-tradarr} sslmode=disable"
JWT_SECRET: ${JWT_SECRET:?JWT_SECRET is required} JWT_SECRET: ${JWT_SECRET:?JWT_SECRET is required}
ENCRYPTION_KEY: ${ENCRYPTION_KEY:?ENCRYPTION_KEY must be 32 bytes hex} ENCRYPTION_KEY: ${ENCRYPTION_KEY:?ENCRYPTION_KEY must be 32 bytes hex}
PORT: "8080" PORT: "8080"
SCRAPER_URL: "http://scraper:3001"
ADMIN_EMAIL: ${ADMIN_EMAIL:-admin@tradarr.local} ADMIN_EMAIL: ${ADMIN_EMAIL:-admin@tradarr.local}
ADMIN_PASSWORD: ${ADMIN_PASSWORD:-changeme} ADMIN_PASSWORD: ${ADMIN_PASSWORD:-changeme}
volumes:
- /home/anthony/.claude:/root/.claude
expose: expose:
- "8080" - "8080"

View File

@ -1,10 +1,18 @@
FROM node:22-alpine AS builder FROM node:22 AS builder
RUN apt-get update && apt-get install -y --no-install-recommends librsvg2-bin && rm -rf /var/lib/apt/lists/*
WORKDIR /app WORKDIR /app
COPY package.json package-lock.json* ./ COPY package.json package-lock.json* ./
RUN npm ci RUN npm ci
COPY . . COPY . .
# Generate PNG icons from SVG source
RUN mkdir -p public/icons && \
rsvg-convert -w 192 -h 192 public/icon-source.svg -o public/icons/icon-192.png && \
rsvg-convert -w 512 -h 512 public/icon-source.svg -o public/icons/icon-512.png
RUN npm run build RUN npm run build
FROM nginx:alpine FROM nginx:alpine

View File

@ -0,0 +1,13 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512">
<!-- Background -->
<rect width="512" height="512" rx="96" fill="#0f172a"/>
<!-- Chart line scaled to center safe zone (~360px) -->
<g transform="translate(76, 76) scale(15)">
<polyline points="22 7 13.5 15.5 8.5 10.5 2 17"
fill="none" stroke="#3b82f6" stroke-width="2.2"
stroke-linecap="round" stroke-linejoin="round"/>
<polyline points="16 7 22 7 22 13"
fill="none" stroke="#3b82f6" stroke-width="2.2"
stroke-linecap="round" stroke-linejoin="round"/>
</g>
</svg>

After

Width:  |  Height:  |  Size: 570 B

View File

@ -4,6 +4,12 @@ export interface AIProvider {
id: string; name: string; model: string; endpoint: string id: string; name: string; model: string; endpoint: string
is_active: boolean; has_key: boolean is_active: boolean; has_key: boolean
} }
export interface AIRoleConfig { provider_id: string; model: string }
export interface AIRoles { summary: AIRoleConfig; report: AIRoleConfig; filter: AIRoleConfig }
export interface OllamaModelInfo {
name: string; size: number; modified_at: string
details: { parameter_size: string; quantization_level: string; family: string }
}
export interface Source { id: string; name: string; type: string; enabled: boolean } export interface Source { id: string; name: string; type: string; enabled: boolean }
export interface ScrapeJob { export interface ScrapeJob {
id: string; source_id: string; source_name: string; status: string id: string; source_id: string; source_name: string; status: string
@ -12,6 +18,7 @@ export interface ScrapeJob {
articles_found: number; error_msg: string; created_at: string articles_found: number; error_msg: string; created_at: string
} }
export interface Setting { key: string; value: string } export interface Setting { key: string; value: string }
export interface ScheduleSlot { id?: string; day_of_week: number; hour: number; minute: number }
export interface AdminUser { id: string; email: string; role: string; created_at: string } export interface AdminUser { id: string; email: string; role: string; created_at: string }
export interface Credential { source_id: string; source_name: string; username: string; has_password: boolean } export interface Credential { source_id: string; source_name: string; username: string; has_password: boolean }
@ -30,6 +37,17 @@ export const adminApi = {
activateProvider: (id: string) => api.post<void>(`/admin/ai-providers/${id}/activate`), activateProvider: (id: string) => api.post<void>(`/admin/ai-providers/${id}/activate`),
deleteProvider: (id: string) => api.delete<void>(`/admin/ai-providers/${id}`), deleteProvider: (id: string) => api.delete<void>(`/admin/ai-providers/${id}`),
listModels: (id: string) => api.get<string[]>(`/admin/ai-providers/${id}/models`), listModels: (id: string) => api.get<string[]>(`/admin/ai-providers/${id}/models`),
probeModels: (data: { name: string; api_key?: string; endpoint?: string }) =>
api.post<string[]>('/admin/ai-providers/probe-models', data),
// AI Roles
getRoles: () => api.get<AIRoles>('/admin/ai-roles'),
updateRole: (role: string, data: AIRoleConfig) => api.put<void>(`/admin/ai-roles/${role}`, data),
// Ollama model management
listOllamaModels: () => api.get<OllamaModelInfo[]>('/admin/ollama/models'),
pullOllamaModel: (name: string) => api.post<void>('/admin/ollama/pull', { name }),
deleteOllamaModel: (name: string) => api.delete<void>(`/admin/ollama/models/${encodeURIComponent(name)}`),
// Sources // Sources
listSources: () => api.get<Source[]>('/admin/sources'), listSources: () => api.get<Source[]>('/admin/sources'),
@ -44,6 +62,10 @@ export const adminApi = {
updateSettings: (settings: Setting[]) => api.put<void>('/admin/settings', { settings }), updateSettings: (settings: Setting[]) => api.put<void>('/admin/settings', { settings }),
getDefaultPrompt: () => api.get<{ prompt: string }>('/admin/settings/default-prompt'), getDefaultPrompt: () => api.get<{ prompt: string }>('/admin/settings/default-prompt'),
// Schedule
getSchedule: () => api.get<ScheduleSlot[]>('/admin/schedule'),
updateSchedule: (slots: ScheduleSlot[]) => api.put<void>('/admin/schedule', { slots }),
// Users // Users
listUsers: () => api.get<AdminUser[]>('/admin/users'), listUsers: () => api.get<AdminUser[]>('/admin/users'),
updateUser: (id: string, email: string, role: string) => updateUser: (id: string, email: string, role: string) =>

View File

@ -14,6 +14,12 @@ async function request<T>(path: string, options: RequestInit = {}): Promise<T> {
...options.headers, ...options.headers,
}, },
}) })
if (res.status === 401) {
localStorage.removeItem('token')
localStorage.removeItem('user')
window.location.href = '/login'
throw new Error('Session expirée')
}
if (!res.ok) { if (!res.ok) {
const err = await res.json().catch(() => ({ error: res.statusText })) const err = await res.json().catch(() => ({ error: res.statusText }))
throw new Error(err.error || res.statusText) throw new Error(err.error || res.statusText)

View File

@ -0,0 +1,20 @@
import { api } from './client'
export interface Report {
id: string
user_id: string
summary_id: string | null
context_excerpt: string
question: string
answer: string
status: 'generating' | 'done' | 'error'
error_msg: string
created_at: string
}
export const reportsApi = {
list: () => api.get<Report[]>('/reports'),
create: (data: { summary_id?: string; excerpts: string[]; question: string }) =>
api.post<Report>('/reports', data),
delete: (id: string) => api.delete<void>(`/reports/${id}`),
}

View File

@ -11,4 +11,5 @@ export interface Summary {
export const summariesApi = { export const summariesApi = {
list: (limit = 10) => api.get<Summary[]>(`/summaries?limit=${limit}`), list: (limit = 10) => api.get<Summary[]>(`/summaries?limit=${limit}`),
generate: () => api.post<Summary>('/summaries/generate'), generate: () => api.post<Summary>('/summaries/generate'),
status: () => api.get<{ generating: boolean }>('/summaries/status'),
} }

View File

@ -1,11 +1,12 @@
import { NavLink } from 'react-router-dom' import { NavLink } from 'react-router-dom'
import { LayoutDashboard, Newspaper, Star, Settings } from 'lucide-react' import { LayoutDashboard, Newspaper, Star, Settings, FileText } from 'lucide-react'
import { cn } from '@/lib/cn' import { cn } from '@/lib/cn'
const items = [ const items = [
{ to: '/', icon: LayoutDashboard, label: 'Dashboard' }, { to: '/', icon: LayoutDashboard, label: 'Dashboard' },
{ to: '/feed', icon: Newspaper, label: 'Actus' }, { to: '/feed', icon: Newspaper, label: 'Actus' },
{ to: '/watchlist', icon: Star, label: 'Watchlist' }, { to: '/watchlist', icon: Star, label: 'Watchlist' },
{ to: '/reports', icon: FileText, label: 'Rapports' },
{ to: '/admin', icon: Settings, label: 'Admin' }, { to: '/admin', icon: Settings, label: 'Admin' },
] ]

View File

@ -1,5 +1,5 @@
import { NavLink } from 'react-router-dom' import { NavLink } from 'react-router-dom'
import { LayoutDashboard, Newspaper, Star, Settings, Key, Cpu, Database, ClipboardList, Users, LogOut, TrendingUp } from 'lucide-react' import { LayoutDashboard, Newspaper, Star, Settings, Key, Cpu, Database, ClipboardList, Users, LogOut, TrendingUp, CalendarDays, FileText } from 'lucide-react'
import { useAuth } from '@/lib/auth' import { useAuth } from '@/lib/auth'
import { cn } from '@/lib/cn' import { cn } from '@/lib/cn'
@ -7,6 +7,7 @@ const navItems = [
{ to: '/', icon: LayoutDashboard, label: 'Dashboard' }, { to: '/', icon: LayoutDashboard, label: 'Dashboard' },
{ to: '/feed', icon: Newspaper, label: 'Actualités' }, { to: '/feed', icon: Newspaper, label: 'Actualités' },
{ to: '/watchlist', icon: Star, label: 'Watchlist' }, { to: '/watchlist', icon: Star, label: 'Watchlist' },
{ to: '/reports', icon: FileText, label: 'Rapports' },
] ]
const adminItems = [ const adminItems = [
@ -15,6 +16,7 @@ const adminItems = [
{ to: '/admin/sources', icon: Database, label: 'Sources' }, { to: '/admin/sources', icon: Database, label: 'Sources' },
{ to: '/admin/jobs', icon: ClipboardList, label: 'Jobs' }, { to: '/admin/jobs', icon: ClipboardList, label: 'Jobs' },
{ to: '/admin/users', icon: Users, label: 'Utilisateurs' }, { to: '/admin/users', icon: Users, label: 'Utilisateurs' },
{ to: '/admin/schedule', icon: CalendarDays, label: 'Planning' },
{ to: '/admin/settings', icon: Settings, label: 'Paramètres' }, { to: '/admin/settings', icon: Settings, label: 'Paramètres' },
] ]

View File

@ -0,0 +1,28 @@
import React from 'react'
function renderInline(text: string): React.ReactNode {
const parts = text.split(/(\*\*[^*]+\*\*)/g)
return parts.map((part, i) =>
part.startsWith('**') && part.endsWith('**')
? <strong key={i}>{part.slice(2, -2)}</strong>
: part
)
}
export function Markdown({ content, className }: { content: string; className?: string }) {
const lines = content.split('\n')
return (
<div className={`space-y-1 text-sm leading-relaxed select-text ${className ?? ''}`}>
{lines.map((line, i) => {
if (line.startsWith('##### ')) return <h5 key={i} className="text-xs font-semibold uppercase tracking-wide text-muted-foreground mt-3">{renderInline(line.slice(6))}</h5>
if (line.startsWith('#### ')) return <h4 key={i} className="text-sm font-semibold mt-3">{renderInline(line.slice(5))}</h4>
if (line.startsWith('### ')) return <h3 key={i} className="text-sm font-semibold text-primary mt-4">{renderInline(line.slice(4))}</h3>
if (line.startsWith('## ')) return <h2 key={i} className="text-base font-bold mt-5 first:mt-0 border-b pb-1">{renderInline(line.slice(3))}</h2>
if (line.startsWith('# ')) return <h1 key={i} className="text-lg font-bold mt-5 first:mt-0">{renderInline(line.slice(2))}</h1>
if (line.startsWith('- ') || line.startsWith('* ')) return <li key={i} className="ml-4 text-muted-foreground list-disc">{renderInline(line.slice(2))}</li>
if (line.trim() === '') return <div key={i} className="h-2" />
return <p key={i} className="text-muted-foreground">{renderInline(line)}</p>
})}
</div>
)
}

View File

@ -37,3 +37,8 @@
::-webkit-scrollbar { width: 6px; height: 6px; } ::-webkit-scrollbar { width: 6px; height: 6px; }
::-webkit-scrollbar-track { @apply bg-muted; } ::-webkit-scrollbar-track { @apply bg-muted; }
::-webkit-scrollbar-thumb { @apply bg-border rounded-full; } ::-webkit-scrollbar-thumb { @apply bg-border rounded-full; }
@layer utilities {
.scrollbar-none { scrollbar-width: none; }
.scrollbar-none::-webkit-scrollbar { display: none; }
}

View File

@ -11,6 +11,8 @@ import { Sources } from '@/pages/admin/Sources'
import { Jobs } from '@/pages/admin/Jobs' import { Jobs } from '@/pages/admin/Jobs'
import { AdminUsers } from '@/pages/admin/AdminUsers' import { AdminUsers } from '@/pages/admin/AdminUsers'
import { AdminSettings } from '@/pages/admin/AdminSettings' import { AdminSettings } from '@/pages/admin/AdminSettings'
import { Schedule } from '@/pages/admin/Schedule'
import { Reports } from '@/pages/Reports'
export const router = createBrowserRouter([ export const router = createBrowserRouter([
{ path: '/login', element: <Login /> }, { path: '/login', element: <Login /> },
@ -20,6 +22,7 @@ export const router = createBrowserRouter([
{ path: '/', element: <Dashboard /> }, { path: '/', element: <Dashboard /> },
{ path: '/feed', element: <Feed /> }, { path: '/feed', element: <Feed /> },
{ path: '/watchlist', element: <Watchlist /> }, { path: '/watchlist', element: <Watchlist /> },
{ path: '/reports', element: <Reports /> },
{ {
path: '/admin', path: '/admin',
element: <AdminLayout />, element: <AdminLayout />,
@ -31,6 +34,7 @@ export const router = createBrowserRouter([
{ path: 'jobs', element: <Jobs /> }, { path: 'jobs', element: <Jobs /> },
{ path: 'users', element: <AdminUsers /> }, { path: 'users', element: <AdminUsers /> },
{ path: 'settings', element: <AdminSettings /> }, { path: 'settings', element: <AdminSettings /> },
{ path: 'schedule', element: <Schedule /> },
], ],
}, },
], ],

View File

@ -1,29 +1,152 @@
import { useState, useEffect } from 'react' import { useState, useEffect, useRef, useCallback } from 'react'
import { TrendingUp, Clock, Sparkles } from 'lucide-react' import { TrendingUp, Clock, Sparkles, MessageSquarePlus, Loader2, Plus, X, Send, ChevronDown, ChevronUp } from 'lucide-react'
import { summariesApi, type Summary } from '@/api/summaries' import { summariesApi, type Summary } from '@/api/summaries'
import { reportsApi } from '@/api/reports'
import { assetsApi, type Asset } from '@/api/assets' import { assetsApi, type Asset } from '@/api/assets'
import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card' import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card'
import { Markdown } from '@/components/ui/markdown'
import { Button } from '@/components/ui/button' import { Button } from '@/components/ui/button'
import { Badge } from '@/components/ui/badge' import { Badge } from '@/components/ui/badge'
import { Spinner } from '@/components/ui/spinner' import { Spinner } from '@/components/ui/spinner'
import { useAuth } from '@/lib/auth' import { useAuth } from '@/lib/auth'
function SummaryContent({ content }: { content: string }) { // ── Text-selection floating button ─────────────────────────────────────────
const lines = content.split('\n')
function useTextSelection(containerRef: React.RefObject<HTMLElement>) {
const [selection, setSelection] = useState<{ text: string; x: number; y: number } | null>(null)
useEffect(() => {
function readSelection() {
const sel = window.getSelection()
const text = sel?.toString().trim()
if (!text || !containerRef.current) { setSelection(null); return }
const range = sel!.getRangeAt(0)
const rect = range.getBoundingClientRect()
const containerRect = containerRef.current.getBoundingClientRect()
setSelection({
text,
x: rect.left - containerRect.left + rect.width / 2,
y: rect.top - containerRect.top - 8,
})
}
function onMouseUp() { readSelection() }
// On mobile, selectionchange fires after the user lifts their finger
function onSelectionChange() {
const sel = window.getSelection()
if (!sel?.toString().trim()) setSelection(null)
else readSelection()
}
function onPointerDown(e: PointerEvent) {
if (!(e.target as Element).closest('[data-context-action]')) setSelection(null)
}
document.addEventListener('mouseup', onMouseUp)
document.addEventListener('selectionchange', onSelectionChange)
document.addEventListener('pointerdown', onPointerDown)
return () => {
document.removeEventListener('mouseup', onMouseUp)
document.removeEventListener('selectionchange', onSelectionChange)
document.removeEventListener('pointerdown', onPointerDown)
}
}, [containerRef])
return selection
}
// ── Context panel (extraits + question) ────────────────────────────────────
function ContextPanel({
excerpts,
onRemove,
onClear,
onSubmit,
}: {
excerpts: string[]
onRemove: (i: number) => void
onClear: () => void
onSubmit: (question: string) => Promise<void>
}) {
const [question, setQuestion] = useState('')
const [submitting, setSubmitting] = useState(false)
const [submitted, setSubmitted] = useState(false)
const [showExcerpts, setShowExcerpts] = useState(false)
async function submit() {
if (!question.trim() || submitting) return
setSubmitting(true)
try {
await onSubmit(question)
setSubmitted(true)
setQuestion('')
setTimeout(() => setSubmitted(false), 2000)
} finally {
setSubmitting(false)
}
}
return ( return (
<div className="space-y-2 text-sm leading-relaxed"> <div className="fixed bottom-16 left-0 right-0 z-40 flex flex-col bg-card border-t shadow-2xl overflow-hidden md:bottom-4 md:left-auto md:right-4 md:w-96 md:rounded-xl md:border">
{lines.map((line, i) => { {/* Header */}
if (line.startsWith('## ')) return <h2 key={i} className="text-base font-semibold mt-4 first:mt-0">{line.slice(3)}</h2> <div className="flex items-center justify-between px-4 py-3 border-b bg-primary/5">
if (line.startsWith('### ')) return <h3 key={i} className="font-medium mt-3">{line.slice(4)}</h3> <button
if (line.startsWith('- ')) return <li key={i} className="ml-4 text-muted-foreground">{line.slice(2)}</li> className="text-sm font-semibold flex items-center gap-2 hover:text-primary transition-colors"
if (line.startsWith('**') && line.endsWith('**')) return <p key={i} className="font-semibold">{line.slice(2, -2)}</p> onClick={() => setShowExcerpts(v => !v)}
if (line.trim() === '') return <div key={i} className="h-1" /> >
return <p key={i} className="text-muted-foreground">{line}</p> <MessageSquarePlus className="h-4 w-4 text-primary" />
})} Contexte ({excerpts.length} extrait{excerpts.length > 1 ? 's' : ''})
<span className="md:hidden">{showExcerpts ? <ChevronDown className="h-3 w-3" /> : <ChevronUp className="h-3 w-3" />}</span>
</button>
<button onClick={onClear} className="text-xs text-muted-foreground hover:text-destructive transition-colors">
Tout effacer
</button>
</div>
{/* Extraits — toujours visible sur desktop, togglable sur mobile */}
<div className={`flex-1 overflow-y-auto px-4 py-3 space-y-2 min-h-0 max-h-[25vh] ${showExcerpts ? 'block' : 'hidden'} md:block`}>
{excerpts.map((e, i) => (
<div key={i} className="flex items-start gap-2 group">
<div className="flex-1 rounded bg-muted/60 px-2 py-1.5 text-xs text-muted-foreground italic line-clamp-3">
«&nbsp;{e}&nbsp;»
</div>
<button
onClick={() => onRemove(i)}
className="mt-1 shrink-0 text-muted-foreground hover:text-destructive md:opacity-0 md:group-hover:opacity-100 transition-opacity"
>
<X className="h-3 w-3" />
</button>
</div>
))}
</div>
{/* Question */}
<div className="border-t px-4 py-3 space-y-2">
<textarea
className="w-full rounded border border-input bg-background px-3 py-2 text-sm focus:outline-none focus:ring-1 focus:ring-ring resize-none"
rows={2}
placeholder="Votre question…"
value={question}
onChange={e => setQuestion(e.target.value)}
onKeyDown={e => { if (e.key === 'Enter' && (e.ctrlKey || e.metaKey)) submit() }}
/>
<Button size="sm" className="w-full" onClick={submit} disabled={!question.trim() || submitting || submitted}>
{submitting
? <><Loader2 className="h-3 w-3 animate-spin" /> Envoi</>
: submitted
? 'Envoyé !'
: <><Send className="h-3 w-3" /> Envoyer (Ctrl+Entrée)</>}
</Button>
</div>
</div> </div>
) )
} }
// ── Summary content renderer ────────────────────────────────────────────────
function SummaryContent({ content }: { content: string }) {
return <Markdown content={content} />
}
// ── Dashboard ───────────────────────────────────────────────────────────────
export function Dashboard() { export function Dashboard() {
const { user } = useAuth() const { user } = useAuth()
const [summaries, setSummaries] = useState<Summary[]>([]) const [summaries, setSummaries] = useState<Summary[]>([])
@ -31,9 +154,26 @@ export function Dashboard() {
const [loading, setLoading] = useState(true) const [loading, setLoading] = useState(true)
const [generating, setGenerating] = useState(false) const [generating, setGenerating] = useState(false)
const [current, setCurrent] = useState<Summary | null>(null) const [current, setCurrent] = useState<Summary | null>(null)
const [excerpts, setExcerpts] = useState<string[]>([])
const summaryRef = useRef<HTMLDivElement>(null)
const textSel = useTextSelection(summaryRef as React.RefObject<HTMLElement>)
useEffect(() => { load() }, []) useEffect(() => { load() }, [])
// Poll generating status every 5s
useEffect(() => {
const interval = setInterval(async () => {
try {
const status = await summariesApi.status()
const wasGenerating = generating
setGenerating(status.generating)
if (wasGenerating && !status.generating) load()
} catch { /* ignore */ }
}, 5000)
return () => clearInterval(interval)
}, [generating])
async function load() { async function load() {
setLoading(true) setLoading(true)
try { try {
@ -55,8 +195,31 @@ export function Dashboard() {
} finally { setGenerating(false) } } finally { setGenerating(false) }
} }
const addExcerpt = useCallback(() => {
if (!textSel) return
setExcerpts(prev => [...prev, textSel.text])
window.getSelection()?.removeAllRanges()
}, [textSel])
async function submitQuestion(question: string) {
await reportsApi.create({
summary_id: current?.id,
excerpts,
question,
})
setExcerpts([])
}
return ( return (
<div className="p-4 md:p-6 space-y-6"> <div className="p-4 md:p-6 space-y-6">
{/* Generating banner */}
{generating && (
<div className="flex items-center gap-3 rounded-md bg-primary/10 border border-primary/20 px-4 py-3 text-sm text-primary">
<Loader2 className="h-4 w-4 animate-spin shrink-0" />
Génération d'un résumé IA en cours — cela peut prendre plusieurs minutes…
</div>
)}
{/* Header */} {/* Header */}
<div className="flex flex-wrap items-start justify-between gap-4"> <div className="flex flex-wrap items-start justify-between gap-4">
<div> <div>
@ -105,7 +268,37 @@ export function Dashboard() {
</div> </div>
</CardHeader> </CardHeader>
<CardContent> <CardContent>
<div ref={summaryRef} className="relative">
<SummaryContent content={current.content} /> <SummaryContent content={current.content} />
{/* Desktop: floating button above selection */}
{textSel && (
<button
data-context-action
onClick={addExcerpt}
className="absolute z-10 hidden md:flex items-center gap-1 rounded-full bg-primary text-primary-foreground text-xs px-2 py-1 shadow-lg hover:bg-primary/90 transition-colors"
style={{ left: textSel.x, top: textSel.y, transform: 'translate(-50%, -100%)' }}
>
<Plus className="h-3 w-3" />
Ajouter au contexte
</button>
)}
</div>
{/* Mobile: fixed bottom bar — avoids conflict with native selection menu */}
{textSel && (
<div className="md:hidden fixed bottom-16 left-0 right-0 z-50 flex justify-center px-4 pointer-events-none">
<button
data-context-action
onClick={addExcerpt}
className="pointer-events-auto flex items-center gap-2 rounded-full bg-primary text-primary-foreground text-sm px-4 py-2.5 shadow-xl"
>
<Plus className="h-4 w-4" />
Ajouter au contexte
</button>
</div>
)}
<p className="text-xs text-muted-foreground mt-4 italic">
Sélectionnez du texte pour l'ajouter au contexte, puis posez une question dans le panneau qui apparaît.
</p>
</CardContent> </CardContent>
</Card> </Card>
) : ( ) : (
@ -141,6 +334,16 @@ export function Dashboard() {
</div> </div>
)} )}
</div> </div>
{/* Panneau contexte flottant */}
{excerpts.length > 0 && (
<ContextPanel
excerpts={excerpts}
onRemove={i => setExcerpts(prev => prev.filter((_, idx) => idx !== i))}
onClear={() => setExcerpts([])}
onSubmit={submitQuestion}
/>
)}
</div> </div>
) )
} }

View File

@ -0,0 +1,128 @@
import { useState, useEffect, useCallback } from 'react'
import { Trash2, FileText, Clock, Loader2, XCircle, AlertCircle } from 'lucide-react'
import { reportsApi, type Report } from '@/api/reports'
import { Card, CardContent } from '@/components/ui/card'
import { Markdown } from '@/components/ui/markdown'
import { Button } from '@/components/ui/button'
import { Spinner } from '@/components/ui/spinner'
import { Badge } from '@/components/ui/badge'
function StatusBadge({ status }: { status: Report['status'] }) {
if (status === 'generating') return (
<Badge variant="secondary" className="gap-1 text-xs">
<Loader2 className="h-3 w-3 animate-spin" /> En cours
</Badge>
)
if (status === 'error') return (
<Badge variant="destructive" className="gap-1 text-xs">
<AlertCircle className="h-3 w-3" /> Erreur
</Badge>
)
return null
}
export function Reports() {
const [reports, setReports] = useState<Report[]>([])
const [loading, setLoading] = useState(true)
const load = useCallback(async () => {
try { setReports((await reportsApi.list()) ?? []) }
finally { setLoading(false) }
}, [])
useEffect(() => { load() }, [load])
// Poll toutes les 3s tant qu'il y a des rapports en cours
useEffect(() => {
const hasGenerating = reports.some(r => r.status === 'generating')
if (!hasGenerating) return
const interval = setInterval(load, 3000)
return () => clearInterval(interval)
}, [reports, load])
async function remove(id: string) {
await reportsApi.delete(id)
setReports(prev => prev.filter(r => r.id !== id))
}
if (loading) return <div className="flex justify-center py-20"><Spinner /></div>
return (
<div className="p-4 md:p-6 space-y-6">
<div>
<h1 className="text-2xl font-bold">Rapports IA</h1>
<p className="text-muted-foreground text-sm">
Vos questions posées sur des extraits de résumés, avec les réponses de l'IA.
</p>
</div>
{reports.length === 0 ? (
<Card className="border-dashed">
<CardContent className="py-12 text-center text-muted-foreground">
<FileText className="h-8 w-8 mx-auto mb-3 opacity-50" />
<p>Aucun rapport enregistré</p>
<p className="text-xs mt-1">Sélectionnez du texte dans un résumé et posez une question à l'IA.</p>
</CardContent>
</Card>
) : (
<div className="space-y-4">
{reports.map(r => (
<Card key={r.id} className={r.status === 'generating' ? 'border-primary/30' : ''}>
<CardContent className="pt-4 pb-4 space-y-3">
{/* Meta */}
<div className="flex items-start justify-between gap-4">
<div className="flex items-center gap-2 flex-wrap">
<div className="flex items-center gap-1 text-xs text-muted-foreground">
<Clock className="h-3 w-3" />
{new Date(r.created_at).toLocaleString('fr-FR')}
</div>
<StatusBadge status={r.status} />
</div>
<Button
variant="ghost"
size="sm"
className="h-6 w-6 p-0 text-muted-foreground hover:text-destructive shrink-0"
onClick={() => remove(r.id)}
title={r.status === 'generating' ? 'Annuler' : 'Supprimer'}
>
{r.status === 'generating'
? <XCircle className="h-3 w-3" />
: <Trash2 className="h-3 w-3" />}
</Button>
</div>
{/* Extraits de contexte */}
{r.context_excerpt.split('\n\n---\n\n').map((excerpt, i) => (
<div key={i} className="rounded bg-muted/60 px-3 py-2 text-xs text-muted-foreground italic">
«&nbsp;{excerpt}&nbsp;»
</div>
))}
{/* Question */}
<p className="text-sm font-medium">{r.question}</p>
{/* Réponse */}
{r.status === 'generating' && (
<div className="flex items-center gap-2 text-sm text-muted-foreground">
<Loader2 className="h-4 w-4 animate-spin text-primary" />
L'IA génère la réponse…
</div>
)}
{r.status === 'done' && (
<div className="border-t pt-3">
<Markdown content={r.answer} />
</div>
)}
{r.status === 'error' && (
<div className="text-sm text-destructive border-t pt-3">
Erreur : {r.error_msg || 'Une erreur est survenue lors de la génération.'}
</div>
)}
</CardContent>
</Card>
))}
</div>
)}
</div>
)
}

View File

@ -1,6 +1,6 @@
import { useState, useEffect } from 'react' import { useState, useEffect, useCallback } from 'react'
import { Plus, Trash2, CheckCircle, RefreshCw } from 'lucide-react' import { Plus, Trash2, Star, Download, HardDrive, Cpu, ChevronDown, ChevronUp, X } from 'lucide-react'
import { adminApi, type AIProvider } from '@/api/admin' import { adminApi, type AIProvider, type AIRoles, type OllamaModelInfo } from '@/api/admin'
import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card' import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card'
import { Button } from '@/components/ui/button' import { Button } from '@/components/ui/button'
import { Input } from '@/components/ui/input' import { Input } from '@/components/ui/input'
@ -9,184 +9,644 @@ import { Select } from '@/components/ui/select'
import { Badge } from '@/components/ui/badge' import { Badge } from '@/components/ui/badge'
import { Spinner } from '@/components/ui/spinner' import { Spinner } from '@/components/ui/spinner'
const PROVIDER_NAMES = ['openai', 'anthropic', 'gemini', 'ollama'] as const const PROVIDER_NAMES = ['openai', 'anthropic', 'gemini', 'ollama', 'claudecode'] as const
export function AIProviders() { const CUSTOM_MODELS_KEY = 'ollama_custom_models'
const [providers, setProviders] = useState<AIProvider[]>([])
const [loading, setLoading] = useState(true)
const [showForm, setShowForm] = useState(false)
const [form, setForm] = useState({ name: 'openai', api_key: '', model: '', endpoint: '' })
const isOllama = form.name === 'ollama'
const [models, setModels] = useState<Record<string, string[]>>({})
const [loadingModels, setLoadingModels] = useState<string | null>(null)
const [editId, setEditId] = useState<string | null>(null)
const [saving, setSaving] = useState(false)
const [error, setError] = useState('')
useEffect(() => { load() }, []) const KNOWN_OLLAMA_MODELS: { name: string; family: string; tags: { tag: string; size: string }[] }[] = [
{ name: 'qwen3', family: 'Qwen3', tags: [{ tag: '0.6b', size: '~0.4 GB' }, { tag: '1.7b', size: '~1.1 GB' }, { tag: '4b', size: '~2.6 GB' }, { tag: '8b', size: '~5.2 GB' }, { tag: '14b', size: '~9.3 GB' }, { tag: '32b', size: '~20 GB' }] },
{ name: 'qwen2.5', family: 'Qwen2.5', tags: [{ tag: '0.5b', size: '~0.4 GB' }, { tag: '1.5b', size: '~1 GB' }, { tag: '3b', size: '~2 GB' }, { tag: '7b', size: '~4.7 GB' }, { tag: '14b', size: '~9 GB' }, { tag: '32b', size: '~20 GB' }] },
{ name: 'llama3.2', family: 'Llama 3.2', tags: [{ tag: '1b', size: '~1.3 GB' }, { tag: '3b', size: '~2 GB' }] },
{ name: 'llama3.1', family: 'Llama 3.1', tags: [{ tag: '8b', size: '~4.9 GB' }, { tag: '70b', size: '~40 GB' }] },
{ name: 'gemma3', family: 'Gemma 3', tags: [{ tag: '1b', size: '~0.8 GB' }, { tag: '4b', size: '~3.3 GB' }, { tag: '12b', size: '~8 GB' }, { tag: '27b', size: '~17 GB' }] },
{ name: 'mistral', family: 'Mistral', tags: [{ tag: '7b', size: '~4.1 GB' }] },
{ name: 'phi4', family: 'Phi-4', tags: [{ tag: 'latest', size: '~9 GB' }] },
{ name: 'phi4-mini', family: 'Phi-4 Mini', tags: [{ tag: 'latest', size: '~2.5 GB' }] },
{ name: 'deepseek-r1', family: 'DeepSeek-R1', tags: [{ tag: '1.5b', size: '~1.1 GB' }, { tag: '7b', size: '~4.7 GB' }, { tag: '8b', size: '~4.9 GB' }, { tag: '14b', size: '~9 GB' }, { tag: '32b', size: '~20 GB' }] },
]
async function load() { function formatSize(bytes: number): string {
setLoading(true) if (bytes >= 1e9) return `${(bytes / 1e9).toFixed(1)} GB`
try { setProviders((await adminApi.listProviders()) ?? []) } finally { setLoading(false) } if (bytes >= 1e6) return `${(bytes / 1e6).toFixed(0)} MB`
return `${bytes} B`
}
function loadCustomModels(): string[] {
try { return JSON.parse(localStorage.getItem(CUSTOM_MODELS_KEY) ?? '[]') } catch { return [] }
}
function saveCustomModels(models: string[]) {
localStorage.setItem(CUSTOM_MODELS_KEY, JSON.stringify(models))
}
// ── Confirmation dialog ────────────────────────────────────────────────────
function ConfirmDownloadDialog({ modelName, onConfirm, onCancel }: {
modelName: string
onConfirm: () => void
onCancel: () => void
}) {
return (
<div className="fixed inset-0 z-50 flex items-center justify-center bg-black/50">
<div className="bg-card border rounded-xl shadow-2xl p-6 w-full max-w-sm mx-4 space-y-4">
<div className="flex items-start justify-between">
<div>
<p className="font-semibold">Télécharger ce modèle ?</p>
<p className="text-sm text-muted-foreground mt-1">
<span className="font-mono text-foreground">{modelName}</span> sera téléchargé depuis Ollama Hub.
Le téléchargement peut prendre plusieurs minutes.
</p>
</div>
<button onClick={onCancel} className="text-muted-foreground hover:text-foreground ml-3 shrink-0">
<X className="h-4 w-4" />
</button>
</div>
<div className="flex gap-2 justify-end">
<Button variant="outline" size="sm" onClick={onCancel}>Annuler</Button>
<Button size="sm" onClick={onConfirm}>
<Download className="h-3 w-3 mr-1" /> Télécharger
</Button>
</div>
</div>
</div>
)
}
// ── Shared model catalogue ─────────────────────────────────────────────────
function ModelCatalogue({
installedNames,
pulling,
onPull,
}: {
installedNames: Set<string>
pulling: string | null
onPull: (name: string) => void
}) {
const [customName, setCustomName] = useState('')
const [pendingModel, setPendingModel] = useState<string | null>(null)
const [customModels, setCustomModels] = useState<string[]>(loadCustomModels)
// All model names in the hardcoded catalogue
const knownNames = new Set(
KNOWN_OLLAMA_MODELS.flatMap(f => f.tags.map(t => `${f.name}:${t.tag}`))
)
function requestPull(name: string) {
setPendingModel(name)
} }
async function loadModels(id: string) { function confirmPull() {
setLoadingModels(id) if (!pendingModel) return
// If not in hardcoded catalogue, add to custom list
if (!knownNames.has(pendingModel)) {
const updated = [...new Set([...customModels, pendingModel])]
setCustomModels(updated)
saveCustomModels(updated)
}
onPull(pendingModel)
setPendingModel(null)
}
function submitCustom() {
const name = customName.trim()
if (!name) return
const fullName = name.includes(':') ? name : `${name}:latest`
setCustomName('')
requestPull(fullName)
}
// Families to display: hardcoded + custom entries
const customEntries = customModels.filter(m => !knownNames.has(m))
return (
<>
{pendingModel && (
<ConfirmDownloadDialog
modelName={pendingModel}
onConfirm={confirmPull}
onCancel={() => setPendingModel(null)}
/>
)}
<div className="space-y-4">
{/* Free-text input */}
<div>
<p className="text-xs font-semibold text-muted-foreground mb-1.5">Nom personnalisé</p>
<div className="flex gap-2">
<Input
className="h-8 text-sm font-mono"
placeholder="ex: llama3.2:3b, mymodel:latest…"
value={customName}
onChange={e => setCustomName(e.target.value)}
onKeyDown={e => e.key === 'Enter' && submitCustom()}
disabled={pulling !== null}
/>
<Button size="sm" className="h-8 shrink-0" onClick={submitCustom}
disabled={!customName.trim() || pulling !== null}>
<Download className="h-3 w-3" /> Installer
</Button>
</div>
</div>
{/* Custom models section */}
{customEntries.length > 0 && (
<div>
<p className="text-xs font-semibold text-muted-foreground mb-1.5">Personnalisés</p>
<div className="flex flex-wrap gap-1.5">
{customEntries.map(fullName => {
const installed = installedNames.has(fullName)
const isPulling = pulling === fullName
return <ModelTag key={fullName} fullName={fullName} size="?" installed={installed} isPulling={isPulling} pulling={pulling} onRequest={requestPull} />
})}
</div>
</div>
)}
{/* Hardcoded catalogue */}
<div className="space-y-3">
{KNOWN_OLLAMA_MODELS.map(family => (
<div key={family.name}>
<p className="text-xs font-semibold text-muted-foreground mb-1.5">{family.family}</p>
<div className="flex flex-wrap gap-1.5">
{family.tags.map(({ tag, size }) => {
const fullName = `${family.name}:${tag}`
const installed = installedNames.has(fullName)
const isPulling = pulling === fullName
return <ModelTag key={fullName} fullName={fullName} size={size} installed={installed} isPulling={isPulling} pulling={pulling} onRequest={requestPull} />
})}
</div>
</div>
))}
</div>
</div>
</>
)
}
function ModelTag({ fullName, size, installed, isPulling, pulling, onRequest }: {
fullName: string; size: string
installed: boolean; isPulling: boolean; pulling: string | null
onRequest: (name: string) => void
}) {
return (
<button
type="button"
disabled={installed || pulling !== null}
onClick={() => !installed && !isPulling && onRequest(fullName)}
className={[
'flex items-center gap-1.5 rounded-md border px-2 py-1 text-xs transition-colors',
installed
? 'border-primary/40 bg-primary/5 text-primary cursor-default'
: isPulling
? 'border-border bg-muted text-muted-foreground cursor-wait'
: pulling !== null
? 'border-border bg-background text-muted-foreground opacity-50 cursor-not-allowed'
: 'border-border bg-background hover:border-primary/60 hover:bg-primary/5 cursor-pointer',
].join(' ')}
>
{isPulling
? <><Spinner className="h-2.5 w-2.5" /> Téléchargement</>
: installed
? <><span className="text-[10px]"></span> {fullName}</>
: <><Download className="h-2.5 w-2.5" /> {fullName} {size !== '?' && <span className="text-muted-foreground">{size}</span>}</>
}
</button>
)
}
// ── Cloud model picker (OpenAI / Anthropic / Gemini / …) ──────────────────
function CloudModelPicker({ value, providerName, apiKey, endpoint, onChange }: {
value: string; providerName: string; apiKey: string; endpoint: string
onChange: (model: string) => void
}) {
const [models, setModels] = useState<string[]>([])
const [loading, setLoading] = useState(false)
const [loadError, setLoadError] = useState('')
const noKeyNeeded = providerName === 'claudecode'
useEffect(() => {
if (noKeyNeeded) loadModels()
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [providerName])
async function loadModels() {
if (!noKeyNeeded && !apiKey) { setLoadError('Renseigne la clé API d\'abord'); return }
setLoading(true); setLoadError('')
try { try {
const m = await adminApi.listModels(id) const list = await adminApi.probeModels({ name: providerName, api_key: apiKey, endpoint })
setModels(prev => ({ ...prev, [id]: m })) setModels(list ?? [])
} catch { /* silently ignore */ } finally { setLoadingModels(null) } if ((list ?? []).length === 0) setLoadError('Aucun modèle trouvé')
} } catch (e) {
setLoadError(e instanceof Error ? e.message : 'Erreur')
async function save() { } finally { setLoading(false) }
setSaving(true); setError('')
try {
if (editId) {
await adminApi.updateProvider(editId, form)
} else {
await adminApi.createProvider(form)
}
setShowForm(false); setEditId(null)
setForm({ name: 'openai', api_key: '', model: '', endpoint: '' })
await load()
} catch (e) { setError(e instanceof Error ? e.message : 'Erreur') } finally { setSaving(false) }
}
async function activate(id: string) {
await adminApi.activateProvider(id)
await load()
}
async function remove(id: string) {
if (!confirm('Supprimer ce fournisseur ?')) return
await adminApi.deleteProvider(id)
await load()
}
function startEdit(p: AIProvider) {
setEditId(p.id)
setForm({ name: p.name, api_key: '', model: p.model, endpoint: p.endpoint })
setShowForm(true)
} }
return ( return (
<div className="space-y-6"> <div className="space-y-1.5">
<div className="flex items-center justify-between">
<div>
<h1 className="text-2xl font-bold">Fournisseurs IA</h1>
<p className="text-muted-foreground text-sm">Configurez les fournisseurs IA et sélectionnez le modèle actif</p>
</div>
<Button onClick={() => { setShowForm(true); setEditId(null) }}>
<Plus className="h-4 w-4" /> Ajouter
</Button>
</div>
{showForm && (
<Card>
<CardHeader><CardTitle>{editId ? 'Modifier' : 'Nouveau fournisseur'}</CardTitle></CardHeader>
<CardContent className="space-y-4">
<div className="grid gap-4 sm:grid-cols-2">
<div className="space-y-1">
<Label>Fournisseur</Label>
<Select
value={form.name}
onChange={e => {
const name = e.target.value
setForm(f => ({
...f,
name,
endpoint: name === 'ollama' ? 'http://ollama:11434' : '',
api_key: '',
}))
}}
disabled={!!editId}
>
{PROVIDER_NAMES.map(n => <option key={n} value={n}>{n}</option>)}
</Select>
</div>
{!isOllama && (
<div className="space-y-1">
<Label>Clé API {editId && <span className="text-muted-foreground">(laisser vide pour conserver)</span>}</Label>
<Input type="password" placeholder="sk-..." value={form.api_key} onChange={e => setForm(f => ({ ...f, api_key: e.target.value }))} />
</div>
)}
<div className="space-y-1">
<Label>Modèle</Label>
<Input placeholder="gpt-4o-mini, claude-sonnet-4-6…" value={form.model} onChange={e => setForm(f => ({ ...f, model: e.target.value }))} />
</div>
{isOllama && (
<div className="space-y-1">
<Label>Endpoint Ollama</Label>
<Input value="http://ollama:11434" readOnly className="opacity-60 cursor-not-allowed" />
</div>
)}
</div>
{error && <p className="text-sm text-destructive">{error}</p>}
<div className="flex gap-2"> <div className="flex gap-2">
<Button onClick={save} disabled={saving}>{saving ? <Spinner className="h-4 w-4" /> : 'Enregistrer'}</Button> <Input
<Button variant="outline" onClick={() => { setShowForm(false); setEditId(null) }}>Annuler</Button> placeholder="gpt-4o-mini, claude-sonnet-4-6…"
value={value}
onChange={e => onChange(e.target.value)}
list="cloud-models-list"
className="flex-1"
/>
<Button type="button" variant="outline" size="sm" className="shrink-0" onClick={loadModels} disabled={loading}>
{loading ? <Spinner className="h-3 w-3" /> : 'Charger'}
</Button>
</div> </div>
</CardContent> {models.length > 0 && (
</Card> <datalist id="cloud-models-list">
{models.map(m => <option key={m} value={m} />)}
</datalist>
)} )}
{models.length > 0 && (
{loading ? ( <Select value={value} onChange={e => onChange(e.target.value)}>
<div className="flex justify-center py-12"><Spinner /></div> <option value=""> Sélectionner un modèle </option>
) : ( {models.map(m => <option key={m} value={m}>{m}</option>)}
<div className="space-y-3">
{providers.map(p => (
<Card key={p.id} className={p.is_active ? 'border-primary/50' : ''}>
<CardContent className="flex flex-wrap items-center gap-4 py-4">
<div className="flex-1 min-w-0">
<div className="flex items-center gap-2 flex-wrap">
<span className="font-semibold capitalize">{p.name}</span>
{p.is_active && <Badge variant="default">Actif</Badge>}
{!p.has_key && p.name !== 'ollama' && <Badge variant="outline" className="text-yellow-500 border-yellow-500">Sans clé</Badge>}
</div>
<div className="text-sm text-muted-foreground mt-1 flex gap-4 flex-wrap">
{p.model && <span>Modèle : <strong>{p.model}</strong></span>}
{p.endpoint && <span>Endpoint : {p.endpoint}</span>}
</div>
{/* Dropdown modèles disponibles */}
{models[p.id] && models[p.id].length > 0 && (
<div className="mt-2 space-y-1">
<Label className="text-xs">Choisir un modèle :</Label>
<Select
className="w-full max-w-xs"
value={p.model}
onChange={async e => {
await adminApi.updateProvider(p.id, { name: p.name, model: e.target.value, endpoint: p.endpoint })
await load()
}}
>
{models[p.id].map(m => <option key={m} value={m}>{m}</option>)}
</Select> </Select>
</div>
)} )}
{loadError && <p className="text-xs text-destructive">{loadError}</p>}
</div> </div>
<div className="flex items-center gap-2 flex-wrap"> )
<Button variant="outline" size="sm" onClick={() => loadModels(p.id)} disabled={loadingModels === p.id}> }
{loadingModels === p.id ? <Spinner className="h-3 w-3" /> : <RefreshCw className="h-3 w-3" />}
Modèles // ── Ollama model picker (inside provider form) ─────────────────────────────
</Button>
<Button variant="outline" size="sm" onClick={() => startEdit(p)}>Modifier</Button> function OllamaModelPicker({ value, installedModels, onSelect, onRefresh }: {
{!p.is_active && ( value: string
<Button size="sm" onClick={() => activate(p.id)}> installedModels: OllamaModelInfo[]
<CheckCircle className="h-3 w-3" /> Activer onSelect: (name: string) => void
</Button> onRefresh: () => Promise<OllamaModelInfo[]>
)} }) {
<Button variant="destructive" size="sm" onClick={() => remove(p.id)}> const [showCatalogue, setShowCatalogue] = useState(installedModels.length === 0)
<Trash2 className="h-3 w-3" /> const [pulling, setPulling] = useState<string | null>(null)
</Button> const installedNames = new Set(installedModels.map(m => m.name))
</div>
</CardContent> async function pull(fullName: string) {
</Card> setPulling(fullName)
try {
await adminApi.pullOllamaModel(fullName)
await new Promise<void>(resolve => {
const iv = setInterval(async () => {
try {
const updated = await onRefresh()
if (updated.some(m => m.name === fullName)) { clearInterval(iv); onSelect(fullName); setShowCatalogue(false); resolve() }
} catch { /* ignore */ }
}, 4000)
setTimeout(() => { clearInterval(iv); resolve() }, 1800000)
})
} finally { setPulling(null) }
}
return (
<div className="space-y-2">
<Select value={value} onChange={e => onSelect(e.target.value)}>
<option value=""> Choisir un modèle installé </option>
{installedModels.map(m => (
<option key={m.name} value={m.name}>
{m.name}{m.details.parameter_size ? ` (${m.details.parameter_size})` : ''}
</option>
))} ))}
{providers.length === 0 && ( </Select>
<Card><CardContent className="py-8 text-center text-muted-foreground">Aucun fournisseur configuré</CardContent></Card> <button type="button" className="flex items-center gap-1 text-xs text-primary hover:underline"
)} onClick={() => setShowCatalogue(v => !v)}>
{showCatalogue ? <ChevronUp className="h-3 w-3" /> : <ChevronDown className="h-3 w-3" />}
{showCatalogue ? 'Masquer le catalogue' : 'Installer un nouveau modèle'}
</button>
{showCatalogue && (
<div className="rounded-lg border bg-muted/30 p-3 max-h-72 overflow-y-auto">
<ModelCatalogue installedNames={installedNames} pulling={pulling} onPull={pull} />
</div> </div>
)} )}
</div> </div>
) )
} }
// ── Role Assignment ────────────────────────────────────────────────────────
const ROLE_LABELS: Record<string, { label: string; desc: string }> = {
summary: { label: 'Résumés', desc: 'Génération du résumé quotidien (passe 2)' },
report: { label: 'Rapports', desc: 'Réponses aux questions sur les résumés' },
filter: { label: 'Filtre articles', desc: 'Sélection des articles pertinents (passe 1)' },
}
function RoleCard({ role, providers, currentProviderID, onSave }: {
role: string; providers: AIProvider[]; currentProviderID: string
onSave: (providerID: string) => Promise<void>
}) {
const [providerID, setProviderID] = useState(currentProviderID)
const [saving, setSaving] = useState(false)
const [saved, setSaved] = useState(false)
useEffect(() => { setProviderID(currentProviderID) }, [currentProviderID])
const { label, desc } = ROLE_LABELS[role]
async function save() {
setSaving(true)
try { await onSave(providerID); setSaved(true); setTimeout(() => setSaved(false), 2000) }
finally { setSaving(false) }
}
return (
<div className="grid gap-3 sm:grid-cols-[1fr_auto] items-end">
<div>
<Label className="text-xs font-semibold uppercase tracking-wide text-muted-foreground">{label}</Label>
<p className="text-xs text-muted-foreground mb-2">{desc}</p>
<Select value={providerID} onChange={e => setProviderID(e.target.value)}>
<option value=""> Fournisseur par défaut </option>
{providers.map(p => <option key={p.id} value={p.id}>{p.name}{p.model ? `${p.model}` : ''}</option>)}
</Select>
</div>
<Button size="sm" onClick={save} disabled={saving} className="whitespace-nowrap">
{saving ? <Spinner className="h-3 w-3" /> : saved ? '✓ Sauvegardé' : 'Appliquer'}
</Button>
</div>
)
}
// ── Ollama model manager card ──────────────────────────────────────────────
function OllamaModelsCard({ installedModels, onRefresh, onDelete }: {
installedModels: OllamaModelInfo[]
onRefresh: () => Promise<OllamaModelInfo[]>
onDelete: (name: string) => Promise<void>
}) {
const [showCatalogue, setShowCatalogue] = useState(false)
const [pulling, setPulling] = useState<string | null>(null)
const [deleting, setDeleting] = useState<string | null>(null)
const installedNames = new Set(installedModels.map(m => m.name))
async function pull(fullName: string) {
setPulling(fullName)
try {
await adminApi.pullOllamaModel(fullName)
await new Promise<void>(resolve => {
const iv = setInterval(async () => {
try {
const updated = await onRefresh()
if (updated.some(m => m.name === fullName)) { clearInterval(iv); resolve() }
} catch { /* ignore */ }
}, 4000)
setTimeout(() => { clearInterval(iv); resolve() }, 1800000)
})
} finally { setPulling(null) }
}
async function remove(name: string) {
setDeleting(name)
try { await onDelete(name) } finally { setDeleting(null) }
}
return (
<Card>
<CardHeader>
<div className="flex items-center justify-between">
<CardTitle className="flex items-center gap-2 text-base">
<HardDrive className="h-4 w-4" /> Modèles Ollama
</CardTitle>
<Button variant="outline" size="sm" onClick={() => setShowCatalogue(v => !v)}>
{showCatalogue ? <><ChevronUp className="h-3.5 w-3.5" /> Masquer</> : <><Download className="h-3.5 w-3.5" /> Installer</>}
</Button>
</div>
</CardHeader>
<CardContent className="space-y-4">
<div>
<h3 className="text-sm font-semibold mb-2 flex items-center gap-1.5">
<Cpu className="h-3.5 w-3.5" /> Installés ({installedModels.length})
</h3>
{installedModels.length === 0 ? (
<p className="text-sm text-muted-foreground italic">Aucun modèle installé</p>
) : (
<div className="space-y-1.5">
{installedModels.map(m => (
<div key={m.name} className="flex items-center justify-between rounded-md border bg-muted/30 px-3 py-2">
<div>
<span className="font-mono text-sm font-medium">{m.name}</span>
<span className="ml-3 text-xs text-muted-foreground">
{formatSize(m.size)}
{m.details.parameter_size && <> · {m.details.parameter_size}</>}
{m.details.quantization_level && <> · {m.details.quantization_level}</>}
</span>
</div>
<Button variant="ghost" size="sm"
className="text-destructive hover:text-destructive h-7 w-7 p-0"
onClick={() => remove(m.name)} disabled={deleting === m.name}>
{deleting === m.name ? <Spinner className="h-3 w-3" /> : <Trash2 className="h-3 w-3" />}
</Button>
</div>
))}
</div>
)}
</div>
{showCatalogue && (
<div className="rounded-lg border bg-muted/30 p-4">
<p className="text-sm font-semibold mb-3">Catalogue</p>
<ModelCatalogue installedNames={installedNames} pulling={pulling} onPull={pull} />
</div>
)}
</CardContent>
</Card>
)
}
// ── Main Page ──────────────────────────────────────────────────────────────
export function AIProviders() {
const [providers, setProviders] = useState<AIProvider[]>([])
const [roles, setRoles] = useState<AIRoles | null>(null)
const [ollamaModels, setOllamaModels] = useState<OllamaModelInfo[]>([])
const [loading, setLoading] = useState(true)
const [showForm, setShowForm] = useState(false)
const [form, setForm] = useState({ name: 'openai', api_key: '', model: '', endpoint: '' })
const [editId, setEditId] = useState<string | null>(null)
const [saving, setSaving] = useState(false)
const [error, setError] = useState('')
const isOllamaForm = form.name === 'ollama'
const isClaudeCodeForm = form.name === 'claudecode'
const loadOllamaModels = useCallback(async (): Promise<OllamaModelInfo[]> => {
try {
console.log('[ollama] fetching installed models…')
const m = await adminApi.listOllamaModels()
const list = m ?? []
console.log(`[ollama] ${list.length} model(s):`, list.map(x => x.name))
setOllamaModels(list)
return list
} catch (e) {
console.error('[ollama] error:', e)
return []
}
}, [])
const load = useCallback(async () => {
setLoading(true)
try {
const [p, r] = await Promise.all([adminApi.listProviders(), adminApi.getRoles()])
setProviders(p ?? [])
setRoles(r)
await loadOllamaModels()
} finally { setLoading(false) }
}, [loadOllamaModels])
useEffect(() => { load() }, [load])
async function save() {
setSaving(true); setError('')
try {
if (editId) { await adminApi.updateProvider(editId, form) }
else { await adminApi.createProvider(form) }
setShowForm(false); setEditId(null)
setForm({ name: 'openai', api_key: '', model: '', endpoint: '' })
await load()
} catch (e) { setError(e instanceof Error ? e.message : 'Erreur') }
finally { setSaving(false) }
}
async function setDefault(id: string) { await adminApi.activateProvider(id); await load() }
async function remove(id: string) {
if (!confirm('Supprimer ce fournisseur ?')) return
await adminApi.deleteProvider(id); await load()
}
function startEdit(p: AIProvider) {
setEditId(p.id); setForm({ name: p.name, api_key: '', model: p.model, endpoint: p.endpoint }); setShowForm(true)
}
async function saveRole(role: string, providerID: string) {
await adminApi.updateRole(role, { provider_id: providerID, model: '' }); await load()
}
async function deleteOllamaModel(name: string) {
await adminApi.deleteOllamaModel(name); await loadOllamaModels()
}
return (
<div className="space-y-6">
<div>
<h1 className="text-2xl font-bold">Fournisseurs IA</h1>
<p className="text-muted-foreground text-sm">Configurez les fournisseurs et assignez un modèle à chaque tâche</p>
</div>
{loading ? (
<div className="flex justify-center py-12"><Spinner /></div>
) : (
<>
{/* Role assignments */}
{roles && providers.length > 0 && (
<Card>
<CardHeader><CardTitle className="text-base">Assignation des tâches</CardTitle></CardHeader>
<CardContent className="space-y-5 divide-y">
{(['summary', 'report', 'filter'] as const).map(role => (
<div key={role} className={role !== 'summary' ? 'pt-5' : ''}>
<RoleCard role={role} providers={providers}
currentProviderID={roles[role].provider_id}
onSave={pid => saveRole(role, pid)} />
</div>
))}
</CardContent>
</Card>
)}
{/* Providers list */}
<Card>
<CardHeader>
<div className="flex items-center justify-between">
<CardTitle className="text-base">Fournisseurs configurés</CardTitle>
<Button size="sm" onClick={() => { setShowForm(true); setEditId(null); setForm({ name: 'openai', api_key: '', model: '', endpoint: '' }) }}>
<Plus className="h-4 w-4" /> Ajouter
</Button>
</div>
</CardHeader>
<CardContent className="space-y-3">
{showForm && (
<div className="rounded-lg border p-4 space-y-4 bg-muted/30">
<p className="font-medium text-sm">{editId ? 'Modifier le fournisseur' : 'Nouveau fournisseur'}</p>
<div className="grid gap-3 sm:grid-cols-2">
<div className="space-y-1">
<Label>Fournisseur</Label>
<Select value={form.name} disabled={!!editId}
onChange={e => {
const name = e.target.value
setForm(f => ({ ...f, name, model: '', api_key: '', endpoint: name === 'ollama' ? 'http://ollama:11434' : '' }))
}}>
{PROVIDER_NAMES.map(n => <option key={n} value={n}>{n}</option>)}
</Select>
</div>
{!isOllamaForm && !isClaudeCodeForm && (
<div className="space-y-1">
<Label>Clé API {editId && <span className="text-muted-foreground text-xs">(vide = conserver)</span>}</Label>
<Input type="password" placeholder="sk-…" value={form.api_key} onChange={e => setForm(f => ({ ...f, api_key: e.target.value }))} />
</div>
)}
{isOllamaForm && (
<div className="space-y-1">
<Label>Endpoint</Label>
<Input value={form.endpoint || 'http://ollama:11434'} onChange={e => setForm(f => ({ ...f, endpoint: e.target.value }))} />
</div>
)}
</div>
<div className="space-y-1">
<Label>Modèle par défaut</Label>
{isOllamaForm ? (
<OllamaModelPicker value={form.model} installedModels={ollamaModels}
onSelect={m => setForm(f => ({ ...f, model: m }))}
onRefresh={loadOllamaModels} />
) : (
<CloudModelPicker
value={form.model}
providerName={form.name}
apiKey={form.api_key}
endpoint={form.endpoint}
onChange={m => setForm(f => ({ ...f, model: m }))}
/>
)}
</div>
{error && <p className="text-sm text-destructive">{error}</p>}
<div className="flex gap-2">
<Button size="sm" onClick={save} disabled={saving}>{saving ? <Spinner className="h-4 w-4" /> : 'Enregistrer'}</Button>
<Button size="sm" variant="outline" onClick={() => { setShowForm(false); setEditId(null) }}>Annuler</Button>
</div>
</div>
)}
{providers.length === 0 && !showForm && (
<p className="text-sm text-muted-foreground text-center py-4">Aucun fournisseur configuré</p>
)}
{providers.map(p => (
<div key={p.id} className={`flex flex-wrap items-center gap-3 rounded-md border px-3 py-2.5 ${p.is_active ? 'border-primary/50 bg-primary/5' : ''}`}>
<div className="flex-1 min-w-0">
<div className="flex items-center gap-2 flex-wrap">
<span className="font-semibold capitalize text-sm">{p.name}</span>
{p.is_active && <Badge variant="default" className="text-xs">Défaut</Badge>}
{!p.has_key && p.name !== 'ollama' && p.name !== 'claudecode' && <Badge variant="outline" className="text-yellow-500 border-yellow-500 text-xs">Sans clé</Badge>}
</div>
<div className="text-xs text-muted-foreground flex gap-3 mt-0.5">
{p.model && <span>Modèle : <strong>{p.model}</strong></span>}
{p.endpoint && <span>{p.endpoint}</span>}
</div>
</div>
<div className="flex items-center gap-1.5">
<Button variant="ghost" size="sm" className="h-7 px-2 text-xs" onClick={() => startEdit(p)}>Modifier</Button>
{!p.is_active && (
<Button size="sm" variant="outline" className="h-7 px-2 text-xs" onClick={() => setDefault(p.id)}>
<Star className="h-3 w-3 mr-1" /> Défaut
</Button>
)}
<Button variant="ghost" size="sm" className="h-7 w-7 p-0 text-destructive hover:text-destructive" onClick={() => remove(p.id)}>
<Trash2 className="h-3 w-3" />
</Button>
</div>
</div>
))}
</CardContent>
</Card>
{/* Ollama model manager — always shown so user can install even without provider configured */}
<OllamaModelsCard
installedModels={ollamaModels}
onRefresh={loadOllamaModels}
onDelete={deleteOllamaModel}
/>
</>
)}
</div>
)
}

View File

@ -1,12 +1,47 @@
import { Outlet, Navigate } from 'react-router-dom' import { NavLink, Outlet, Navigate } from 'react-router-dom'
import { Cpu, Key, Database, ClipboardList, Users, CalendarDays, Settings } from 'lucide-react'
import { useAuth } from '@/lib/auth' import { useAuth } from '@/lib/auth'
import { cn } from '@/lib/cn'
const adminItems = [
{ to: '/admin/ai', icon: Cpu, label: 'Fournisseurs IA' },
{ to: '/admin/credentials', icon: Key, label: 'Identifiants' },
{ to: '/admin/sources', icon: Database, label: 'Sources' },
{ to: '/admin/jobs', icon: ClipboardList,label: 'Jobs' },
{ to: '/admin/users', icon: Users, label: 'Utilisateurs' },
{ to: '/admin/schedule', icon: CalendarDays, label: 'Planning' },
{ to: '/admin/settings', icon: Settings, label: 'Paramètres' },
]
export function AdminLayout() { export function AdminLayout() {
const { isAdmin } = useAuth() const { isAdmin } = useAuth()
if (!isAdmin) return <Navigate to="/" replace /> if (!isAdmin) return <Navigate to="/" replace />
return ( return (
<div>
{/* Mobile sub-nav — scrollable horizontal tabs */}
<nav className="md:hidden flex overflow-x-auto border-b bg-card scrollbar-none sticky top-0 z-10">
{adminItems.map(({ to, icon: Icon, label }) => (
<NavLink
key={to}
to={to}
className={({ isActive }) =>
cn(
'flex shrink-0 flex-col items-center gap-1 px-4 py-3 text-xs font-medium transition-colors border-b-2',
isActive
? 'border-primary text-primary'
: 'border-transparent text-muted-foreground'
)
}
>
<Icon className="h-4 w-4" />
{label}
</NavLink>
))}
</nav>
<div className="p-6"> <div className="p-6">
<Outlet /> <Outlet />
</div> </div>
</div>
) )
} }

View File

@ -8,19 +8,27 @@ import { Label } from '@/components/ui/label'
import { Spinner } from '@/components/ui/spinner' import { Spinner } from '@/components/ui/spinner'
const NUMERIC_SETTINGS: Record<string, { label: string; description: string }> = { const NUMERIC_SETTINGS: Record<string, { label: string; description: string }> = {
scrape_interval_minutes: { label: 'Intervalle de scraping (minutes)', description: 'Fréquence de récupération des actualités' }, summary_max_articles: { label: 'Articles max par résumé', description: 'Nombre maximum d\'articles envoyés à l\'IA pour la passe 2 (résumé)' },
articles_lookback_hours: { label: 'Fenêtre d\'analyse (heures)', description: 'Période couverte pour les résumés IA' }, filter_batch_size: { label: 'Taille des batches (filtre)', description: 'Nombre d\'articles par appel IA lors de la passe 1 (filtrage). Réduire pour des réponses plus rapides.' },
summary_max_articles: { label: 'Articles max par résumé', description: 'Nombre maximum d\'articles envoyés à l\'IA' },
} }
const COMMON_TIMEZONES = [
'Europe/Paris', 'Europe/London', 'Europe/Berlin', 'Europe/Madrid', 'Europe/Rome',
'America/New_York', 'America/Chicago', 'America/Los_Angeles',
'Asia/Tokyo', 'Asia/Hong_Kong', 'Asia/Singapore',
'UTC',
]
export function AdminSettings() { export function AdminSettings() {
const [settings, setSettings] = useState<Setting[]>([]) const [settings, setSettings] = useState<Setting[]>([])
const [loading, setLoading] = useState(true) const [loading, setLoading] = useState(true)
const [values, setValues] = useState<Record<string, string>>({}) const [values, setValues] = useState<Record<string, string>>({})
const [defaultPrompt, setDefaultPrompt] = useState('') const [defaultPrompt, setDefaultPrompt] = useState('')
const [saving, setSaving] = useState(false) const [saving, setSaving] = useState(false)
const [savingTz, setSavingTz] = useState(false)
const [savingPrompt, setSavingPrompt] = useState(false) const [savingPrompt, setSavingPrompt] = useState(false)
const [saved, setSaved] = useState(false) const [saved, setSaved] = useState(false)
const [savedTz, setSavedTz] = useState(false)
const [savedPrompt, setSavedPrompt] = useState(false) const [savedPrompt, setSavedPrompt] = useState(false)
useEffect(() => { load() }, []) useEffect(() => { load() }, [])
@ -53,6 +61,13 @@ export function AdminSettings() {
setTimeout(() => setSaved(false), 2000) setTimeout(() => setSaved(false), 2000)
} }
async function saveTimezone() {
setSavingTz(true); setSavedTz(false)
await adminApi.updateSettings([{ key: 'timezone', value: values['timezone'] ?? 'Europe/Paris' }])
setSavingTz(false); setSavedTz(true)
setTimeout(() => setSavedTz(false), 2000)
}
async function savePrompt() { async function savePrompt() {
setSavingPrompt(true); setSavedPrompt(false) setSavingPrompt(true); setSavedPrompt(false)
await adminApi.updateSettings([{ key: 'ai_system_prompt', value: values['ai_system_prompt'] ?? '' }]) await adminApi.updateSettings([{ key: 'ai_system_prompt', value: values['ai_system_prompt'] ?? '' }])
@ -75,6 +90,35 @@ export function AdminSettings() {
<p className="text-muted-foreground text-sm">Configuration globale du service</p> <p className="text-muted-foreground text-sm">Configuration globale du service</p>
</div> </div>
{/* Fuseau horaire */}
<Card>
<CardHeader><CardTitle>Fuseau horaire</CardTitle></CardHeader>
<CardContent className="space-y-4">
<div className="space-y-1">
<Label>Timezone (IANA)</Label>
<p className="text-xs text-muted-foreground">
Utilisé pour le planning de scraping et l'horodatage des résumés
</p>
<div className="flex gap-2 max-w-sm">
<Input
list="tz-list"
value={values['timezone'] ?? 'Europe/Paris'}
onChange={e => setValues(v => ({ ...v, timezone: e.target.value }))}
placeholder="Europe/Paris"
/>
<datalist id="tz-list">
{COMMON_TIMEZONES.map(tz => <option key={tz} value={tz} />)}
</datalist>
</div>
</div>
<Button onClick={saveTimezone} disabled={savingTz}>
{savingTz ? <Spinner className="h-4 w-4" /> : <Save className="h-4 w-4" />}
{savedTz ? 'Enregistré !' : 'Enregistrer'}
</Button>
</CardContent>
</Card>
{/* Paramètres généraux */}
<Card> <Card>
<CardHeader><CardTitle>Paramètres généraux</CardTitle></CardHeader> <CardHeader><CardTitle>Paramètres généraux</CardTitle></CardHeader>
<CardContent className="space-y-6"> <CardContent className="space-y-6">
@ -101,6 +145,7 @@ export function AdminSettings() {
</CardContent> </CardContent>
</Card> </Card>
{/* Contexte IA */}
<Card> <Card>
<CardHeader> <CardHeader>
<CardTitle>Contexte IA</CardTitle> <CardTitle>Contexte IA</CardTitle>

View File

@ -0,0 +1,145 @@
import { useState, useEffect } from 'react'
import { Plus, Trash2, Save } from 'lucide-react'
import { adminApi, type ScheduleSlot } from '@/api/admin'
import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card'
import { Button } from '@/components/ui/button'
import { Spinner } from '@/components/ui/spinner'
const DAYS = [
{ label: 'Lundi', short: 'LUN', value: 1 },
{ label: 'Mardi', short: 'MAR', value: 2 },
{ label: 'Mercredi', short: 'MER', value: 3 },
{ label: 'Jeudi', short: 'JEU', value: 4 },
{ label: 'Vendredi', short: 'VEN', value: 5 },
{ label: 'Samedi', short: 'SAM', value: 6 },
{ label: 'Dimanche', short: 'DIM', value: 0 },
]
type SlotKey = `${number}-${number}-${number}`
function toKey(s: ScheduleSlot): SlotKey {
return `${s.day_of_week}-${s.hour}-${s.minute}`
}
function fmt(h: number, m: number) {
return `${String(h).padStart(2, '0')}:${String(m).padStart(2, '0')}`
}
export function Schedule() {
const [slots, setSlots] = useState<ScheduleSlot[]>([])
const [loading, setLoading] = useState(true)
const [saving, setSaving] = useState(false)
const [saved, setSaved] = useState(false)
const [newTimes, setNewTimes] = useState<Record<number, string>>({})
useEffect(() => { load() }, [])
async function load() {
setLoading(true)
try { setSlots((await adminApi.getSchedule()) ?? []) } finally { setLoading(false) }
}
function slotsForDay(day: number) {
return slots
.filter(s => s.day_of_week === day)
.sort((a, b) => a.hour !== b.hour ? a.hour - b.hour : a.minute - b.minute)
}
function addSlot(day: number) {
const time = newTimes[day] || '06:00'
const [h, m] = time.split(':').map(Number)
const newSlot: ScheduleSlot = { day_of_week: day, hour: h, minute: m }
if (slots.some(s => toKey(s) === toKey(newSlot))) return
setSlots(prev => [...prev, newSlot])
setNewTimes(p => ({ ...p, [day]: '06:00' }))
}
function removeSlot(slot: ScheduleSlot) {
setSlots(prev => prev.filter(s => toKey(s) !== toKey(slot)))
}
async function save() {
setSaving(true); setSaved(false)
await adminApi.updateSchedule(slots)
setSaving(false); setSaved(true)
setTimeout(() => setSaved(false), 2000)
}
if (loading) return <div className="flex justify-center py-20"><Spinner /></div>
return (
<div className="space-y-6">
<div className="flex items-center justify-between">
<div>
<h1 className="text-2xl font-bold">Planning hebdomadaire</h1>
<p className="text-muted-foreground text-sm">
Définissez les créneaux de scraping + résumé IA pour chaque jour
</p>
</div>
<Button onClick={save} disabled={saving}>
{saving ? <Spinner className="h-4 w-4" /> : <Save className="h-4 w-4" />}
{saved ? 'Enregistré !' : 'Enregistrer'}
</Button>
</div>
<div className="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-4 xl:grid-cols-7 gap-3">
{DAYS.map(day => {
const daySlots = slotsForDay(day.value)
const isWeekend = day.value === 0 || day.value === 6
return (
<Card key={day.value} className={isWeekend ? 'border-muted' : ''}>
<CardHeader className="pb-2 pt-4 px-4">
<CardTitle className="text-sm font-semibold">
<span className="hidden xl:block">{day.label}</span>
<span className="xl:hidden">{day.short}</span>
</CardTitle>
</CardHeader>
<CardContent className="px-4 pb-4 space-y-2">
{/* Créneaux existants */}
{daySlots.length === 0 && (
<p className="text-xs text-muted-foreground italic">Aucun créneau</p>
)}
{daySlots.map(slot => (
<div
key={toKey(slot)}
className="flex items-center justify-between rounded bg-primary/10 px-2 py-1"
>
<span className="text-sm font-mono font-medium">
{fmt(slot.hour, slot.minute)}
</span>
<button
onClick={() => removeSlot(slot)}
className="text-muted-foreground hover:text-destructive transition-colors ml-2"
>
<Trash2 className="h-3 w-3" />
</button>
</div>
))}
{/* Ajout d'un créneau */}
<div className="flex items-center gap-1 pt-1">
<input
type="time"
value={newTimes[day.value] ?? '06:00'}
onChange={e => setNewTimes(p => ({ ...p, [day.value]: e.target.value }))}
className="flex-1 min-w-0 rounded border border-input bg-background px-2 py-1 text-xs font-mono focus:outline-none focus:ring-1 focus:ring-ring"
/>
<button
onClick={() => addSlot(day.value)}
className="rounded bg-primary/10 p-1 hover:bg-primary/20 transition-colors"
>
<Plus className="h-3 w-3" />
</button>
</div>
</CardContent>
</Card>
)
})}
</div>
<p className="text-xs text-muted-foreground">
À chaque créneau, le service lance le scraping de toutes les sources actives puis génère les résumés IA.
</p>
</div>
)
}

View File

@ -34,12 +34,12 @@ export function Sources() {
<div className="space-y-3"> <div className="space-y-3">
{sources.map(s => ( {sources.map(s => (
<Card key={s.id}> <Card key={s.id}>
<CardContent className="flex items-center justify-between py-4"> <CardContent className="flex flex-col gap-3 py-4 sm:flex-row sm:items-center sm:justify-between">
<div> <div>
<span className="font-semibold">{s.name}</span> <span className="font-semibold">{s.name}</span>
<span className="ml-2 text-xs text-muted-foreground capitalize">({s.type})</span> <span className="ml-2 text-xs text-muted-foreground capitalize">({s.type})</span>
</div> </div>
<div className="flex items-center gap-3"> <div className="flex items-center gap-2">
<Badge variant={s.enabled ? 'default' : 'outline'}> <Badge variant={s.enabled ? 'default' : 'outline'}>
{s.enabled ? 'Activée' : 'Désactivée'} {s.enabled ? 'Activée' : 'Désactivée'}
</Badge> </Badge>

View File

@ -15,8 +15,7 @@
"noUnusedLocals": true, "noUnusedLocals": true,
"noUnusedParameters": true, "noUnusedParameters": true,
"noFallthroughCasesInSwitch": true, "noFallthroughCasesInSwitch": true,
"baseUrl": ".", "paths": { "@/*": ["./src/*"] }
"paths": { "@/*": ["src/*"] }
}, },
"include": ["src"] "include": ["src"]
} }

View File

@ -16,9 +16,10 @@ export default defineConfig({
theme_color: '#0f172a', theme_color: '#0f172a',
background_color: '#0f172a', background_color: '#0f172a',
display: 'standalone', display: 'standalone',
start_url: '/',
icons: [ icons: [
{ src: '/icons/icon-192.png', sizes: '192x192', type: 'image/png' }, { src: '/icons/icon-192.png', sizes: '192x192', type: 'image/png', purpose: 'any' },
{ src: '/icons/icon-512.png', sizes: '512x512', type: 'image/png' }, { src: '/icons/icon-512.png', sizes: '512x512', type: 'image/png', purpose: 'any maskable' },
], ],
}, },
workbox: { workbox: {

View File

@ -0,0 +1,46 @@
FROM node:20-slim
RUN apt-get update && apt-get install -y \
chromium \
fonts-liberation \
libasound2 \
libatk-bridge2.0-0 \
libatk1.0-0 \
libcairo2 \
libcups2 \
libdbus-1-3 \
libexpat1 \
libfontconfig1 \
libgbm1 \
libglib2.0-0 \
libgtk-3-0 \
libnspr4 \
libnss3 \
libpango-1.0-0 \
libpangocairo-1.0-0 \
libx11-6 \
libx11-xcb1 \
libxcb1 \
libxcomposite1 \
libxcursor1 \
libxdamage1 \
libxext6 \
libxfixes3 \
libxi6 \
libxrandr2 \
libxrender1 \
libxss1 \
libxtst6 \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
ENV PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium
WORKDIR /app
COPY package*.json ./
RUN npm install --omit=dev
COPY . .
EXPOSE 3001
CMD ["node", "index.js"]

205
scraper-service/index.js Normal file
View File

@ -0,0 +1,205 @@
const express = require('express')
const puppeteer = require('puppeteer-extra')
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())
const app = express()
app.use(express.json())
const CHROME_PATH = process.env.PUPPETEER_EXECUTABLE_PATH || '/usr/bin/chromium'
const PORT = process.env.PORT || 3001
function launchBrowser() {
return puppeteer.launch({
executablePath: CHROME_PATH,
headless: true,
args: [
'--no-sandbox',
'--disable-setuid-sandbox',
'--disable-dev-shm-usage',
'--disable-gpu',
'--window-size=1920,1080',
'--disable-blink-features=AutomationControlled',
],
})
}
async function tryClick(page, selectors) {
for (const sel of selectors) {
try {
const el = await page.$(sel)
if (el) { await el.click(); return true }
} catch {}
}
await page.keyboard.press('Enter')
return false
}
async function tryType(page, selectors, text) {
for (const sel of selectors) {
try {
await page.waitForSelector(sel, { timeout: 4000 })
await page.type(sel, text, { delay: 60 })
return true
} catch {}
}
return false
}
app.get('/health', (_, res) => res.json({ ok: true }))
app.post('/bloomberg/scrape', async (req, res) => {
const { username, password } = req.body || {}
if (!username || !password) {
return res.status(400).json({ error: 'username and password required' })
}
let browser
try {
browser = await launchBrowser()
const page = await browser.newPage()
await page.setViewport({ width: 1920, height: 1080 })
// Hide automation signals
await page.evaluateOnNewDocument(() => {
Object.defineProperty(navigator, 'webdriver', { get: () => undefined })
window.chrome = { runtime: {} }
Object.defineProperty(navigator, 'plugins', { get: () => [1, 2, 3] })
})
console.log('[bloomberg] navigating to login page')
await page.goto('https://www.bloomberg.com/account/signin', {
waitUntil: 'networkidle2',
timeout: 60000,
})
await new Promise(r => setTimeout(r, 2000))
// Debug: état de la page avant de chercher l'email
const pageInputs = await page.evaluate(() =>
Array.from(document.querySelectorAll('input')).map(i => ({
type: i.type, name: i.name, id: i.id, placeholder: i.placeholder, visible: i.offsetParent !== null
}))
)
console.log('[bloomberg] inputs on login page:', JSON.stringify(pageInputs))
const pageTitle = await page.title()
console.log('[bloomberg] page title:', pageTitle)
console.log('[bloomberg] entering email')
const emailSelectors = [
'#email-form-input',
'input[id="email-form-input"]',
'input[type="email"]',
'input[name="text-input"]',
'input[placeholder*="email" i]',
]
const emailOk = await tryType(page, emailSelectors, username)
if (!emailOk) throw new Error('could not find email input')
await new Promise(r => setTimeout(r, 800))
// Click submit via JS pour contourner les boutons désactivés
const submitted = await page.evaluate(() => {
const btns = Array.from(document.querySelectorAll('button'))
const btn = btns.find(b =>
b.type === 'submit' ||
/continue|next|sign.?in/i.test(b.textContent)
)
if (btn) { btn.click(); return true }
const form = document.querySelector('form')
if (form) { form.submit(); return true }
return false
})
if (!submitted) await page.keyboard.press('Enter')
// Attendre que la page change (password input apparaît ou navigation)
try {
await page.waitForFunction(
() => document.querySelector('input[type="password"]') !== null,
{ timeout: 10000 }
)
} catch {
await new Promise(r => setTimeout(r, 3000))
}
console.log('[bloomberg] after email submit, url:', page.url())
// Debug inputs disponibles
const allInputs = await page.evaluate(() =>
Array.from(document.querySelectorAll('input')).map(i => ({
type: i.type, name: i.name, id: i.id, placeholder: i.placeholder
}))
)
console.log('[bloomberg] inputs after email submit:', JSON.stringify(allInputs))
console.log('[bloomberg] entering password')
const pwdOk = await tryType(page, [
'input[type="password"]',
'input[name="password"]',
'input[autocomplete="current-password"]',
'input[autocomplete="password"]',
], password)
if (!pwdOk) throw new Error('could not find password input — check logs above for available inputs')
await new Promise(r => setTimeout(r, 500))
await tryClick(page, ['button[type="submit"]', 'input[type="submit"]'])
await new Promise(r => setTimeout(r, 3000))
const currentURL = page.url()
console.log('[bloomberg] after login, url:', currentURL)
const pages = [
'https://www.bloomberg.com/markets',
'https://www.bloomberg.com/technology',
'https://www.bloomberg.com/economics',
]
const articles = []
const seen = new Set()
for (const url of pages) {
try {
console.log('[bloomberg] scraping', url)
await page.goto(url, { waitUntil: 'networkidle2', timeout: 30000 })
await new Promise(r => setTimeout(r, 2000))
const items = await page.evaluate(() => {
const results = []
const seen = new Set()
const links = document.querySelectorAll(
'a[href*="/news/articles"], a[href*="/opinion/"], a[href*="/markets/"]'
)
links.forEach(a => {
if (seen.has(a.href)) return
seen.add(a.href)
const titleEl = a.querySelector('h1,h2,h3,h4,[class*="headline"],[class*="title"]')
const text = titleEl ? titleEl.innerText.trim() : a.innerText.trim()
if (text.length > 20 && a.href.includes('bloomberg.com')) {
results.push({ title: text, url: a.href })
}
})
return results.slice(0, 25)
})
for (const item of items) {
if (!seen.has(item.url) && item.title && item.url) {
seen.add(item.url)
articles.push(item)
}
}
console.log('[bloomberg]', url, '->', items.length, 'articles')
} catch (e) {
console.error('[bloomberg] error on', url, ':', e.message)
}
}
console.log('[bloomberg] total:', articles.length, 'articles')
res.json({ articles })
} catch (e) {
console.error('[bloomberg] scrape error:', e.message)
res.status(500).json({ error: e.message })
} finally {
if (browser) await browser.close()
}
})
app.listen(PORT, () => console.log(`scraper-service listening on :${PORT}`))

View File

@ -0,0 +1,14 @@
{
"name": "tradarr-scraper-service",
"version": "1.0.0",
"main": "index.js",
"scripts": {
"start": "node index.js"
},
"dependencies": {
"express": "^4.19.2",
"puppeteer-extra": "^3.3.6",
"puppeteer-extra-plugin-stealth": "^2.11.2",
"puppeteer": "^22.0.0"
}
}