bot protection

Auf welchem Server läuft ~~searx.space~~ fairsuch?

Auf serles (das war ursprünglich nur eine Videobridge), Liste unserer Server siehe https://git.fairkom.net/operations/infra - dein key ist nun drauf. searx wäre wohl auch ein Kandidat für ein k8s deployment.

Danke. Hab die Dockerfile in /root/filtron angepasst, aber noch nicht gestartet. (nginx muss auch noch umgestellt werden.) Vielleicht sollten searx+filtron in eine docker-compose.yml, damit sie gleichzeitig gestartet werden und in einem eigenen network sind, aber zum Test starte ich es erst mal ohne.

added inactive label

There is a new bot limiter available. On our new k8s setup we had to deploy redis #9 for making it work.

Now some headers and configs are still missing, see https://docs.searxng.org/admin/searx.limiter.html

│ 2024-04-09 19:49:54,187 ERROR:searx.botdetection: X-Forwarded-For header is not set!                                                                                                                            
│ 2024-04-09 19:49:54,187 ERROR:searx.botdetection: X-Real-IP header is not set!                                                                                                                                  
│ 2024-04-09 19:49:54,477 WARNING:searx.botdetection.config: missing config file: /etc/searxng/limiter.toml

We keep the default settings for the limiter https://docs.searxng.org/admin/searx.limiter.html#limiter-toml

Tried in ingress 79decf85 , but it did not do the job:

  nginx.ingress.kubernetes.io/enable-real-ip: true
  nginx.ingress.kubernetes.io/forwarded-for-header: proxy_protocol

@armin.felder where and how do I need to configure ingress for that?

Tried adding in ingress.yaml, but searxng still mourns about missing headers:

---
# ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx
data:
  enable-real-ip: "true"
  forwarded-for-header: "X-Forwarded-For"

sniffing with wireshark on dev instance - all requested headers are here:

Frame 533: 932 bytes on wire (7456 bits), 932 bytes captured (7456 bits) on interface -, id 0
Linux cooked capture v2
Internet Protocol Version 4, Src: 10.41.128.4, Dst: 10.45.0.42
Transmission Control Protocol, Src Port: 56800, Dst Port: 8080, Seq: 1, Ack: 1, Len: 860
Hypertext Transfer Protocol
    POST /search HTTP/1.1\r\n
    Host: fairsuch.dev.osalliance.com\r\n
    X-Request-ID: b208c7baa4fa9e8fecdf018aadcb7654\r\n
    X-Real-IP: 84.115.214.23\r\n
    X-Forwarded-For: 84.115.214.23\r\n
    X-Forwarded-Host: fairsuch.dev.osalliance.com\r\n
    X-Forwarded-Port: 443\r\n
    X-Forwarded-Proto: https\r\n
    X-Forwarded-Scheme: https\r\n
    X-Scheme: https\r\n
    Content-Length: 77\r\n
    user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0\r\n
    accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\r\n
    accept-language: en-US,en;q=0.5\r\n
    accept-encoding: gzip, deflate, br\r\n
    content-type: application/x-www-form-urlencoded\r\n
    origin: null\r\n
    upgrade-insecure-requests: 1\r\n
    sec-fetch-dest: document\r\n
    sec-fetch-mode: navigate\r\n
    sec-fetch-site: same-origin\r\n
    sec-fetch-user: ?1\r\n
    priority: u=1\r\n
    \r\n
    [Full request URI: http://fairsuch.dev.osalliance.com/search]
    [HTTP request 1/1]
    [Response in frame: 1170]
    File Data: 77 bytes
HTML Form URL Encoded: application/x-www-form-urlencoded

bot detection seems to work even when log says that requested X-Real-IP and X-Forwarded-For headers are not set

2024-04-10 18:27:43,710 ERROR:searx.botdetection.ip_limit: BLOCK: too many request from 84.239.45.12/32 in SUSPICIOUS_IP_WINDOW (redirect to /)

so we keep it for the moment as is as detection seems to work

bot protection

Designs

Child items ...

Activity