Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
We Have This Issue With Our Oss Deployment: One Azure Container App For

We have this issue with our OSS deployment: one Azure Container App for web server and one Azure Container App for api server .
The issue: the console log don't display in the WebUI for some of our task (~800kb)
Digging further we found that the console result is corrupted. We extracted the curl command:

curl '
'   --compressed   -X POST   -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0'   -H 'Accept: application/json'   -H 'Accept-Language: en-US,en;q=0.5'   -H 'Accept-Encoding: gzip, deflate, br, zstd'   -H 'X-Allegro-Client: Webapp-2.2.0-690'   -H 'Content-Type: application/json'   -H 'Origin: 
'   -H 'DNT: 1'   -H 'Connection: keep-alive'   -H 'Referer: 
..azurecontainerapps.io/projects/*/tasks/10ba913ad81642faaf5d0e3ff1c01aeb/log?columns=selected^&columns=type^&columns=name^&columns=last_iteration^&columns=parent.name^&order=-last_update^&filter='   -H 'Cookie: clearml_token_basic=REDACTED'   -H 'Sec-Fetch-Dest: empty'   -H 'Sec-Fetch-Mode: cors'   -H 'Sec-Fetch-Site: same-origin'   -H 'Sec-GPC: 1'   -H 'TE: trailers'   --data-raw '{"task":"10ba913ad81642faaf5d0e3ff1c01aeb","batch_size":1000,"navigate_earlier":true}

When running this curl: 8/10 we get curl: (18) HTTP/2 stream 1 was reset With about 56009 received on total of 87025

The log on the web container show:

[error] 13#13: *41888 upstream prematurely closed connection while reading upstream, client: 100.100.1.217, server: _, request: "POST /api/v2.33/events.get_task_log HTTP/1.1", upstream: "
", host: "REDACTED-web.azurecontainerapps.io", referrer: "
"

We though first that the issue was between the api server and our ES server, as suggested here .
But then I tried to query directly the api server but slightly modifying the address:

curl '
'  ....

And I get the full json 100% of the time.

Which means: there are some issue in the communication between api and web
They are 2 Azure Container App running in the same Azure Container App Environment. The log never show any sign of issue between the two a part from this task log.

May be it's an issue in how the web forward data from the api to the client ?? @<1523701070390366208:profile|CostlyOstrich36>

  
  
Posted one month ago
Votes Newest

Answers 16


None

  
  
Posted one month ago

I did.
I am now redeploying to new container to be sure.

  
  
Posted one month ago

can you try running the curl request from within the web container?

  
  
Posted one month ago

yes, I did try that exactly: all curl request succeed

  
  
Posted one month ago

on second thought if direct request to api server works while reverse proxy fails, this points us only to nginx.

  
  
Posted one month ago

@<1523703436166565888:profile|DeterminedCrab71> Found the solution:
We needed proxy_set_header Connection ""; in the location /api section. Putting it above in the server section had no effect

  
  
Posted one month ago

yes thanks, that would be very helpful

  
  
Posted one month ago

Should I raise a github issue ticket ?

  
  
Posted one month ago

i'd focus on nginx or whatever azure infra you have between nginx and the client.

  
  
Posted one month ago

the weird things is if it's a Azure ACA issue, it would be known right ? There are so many people who use ACA and having ACA talking to each other.
this is really weird ...

  
  
Posted one month ago

can you make sure the change to client_max_body_size was applied with nginx -T ?

  
  
Posted one month ago

and I tried the curl against the web url: and it fails 8/10

  
  
Posted one month ago

will send the nginx -T results once the container is deployed

  
  
Posted one month ago

@<1523703436166565888:profile|DeterminedCrab71> Thansk for the suggestion. But no effect.
We already have client_max_body_size 0; in the server section
I tried to set both http and server section 100M but nothing changes.

Do you think the gzip be related ?

  
  
Posted one month ago

Deployed fresh new and ran nginx -T in the container:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
error_log stderr notice;

events {
        worker_connections 768;
        # multi_accept on;
}

http {
        client_max_body_size 100M;
        rewrite_log on;

        log_format devlogging '$remote_addr - $remote_user [$time_local] "proxy_host" $proxy_host "proxy_add_x_forwarded_for" $proxy_add_x_forwarded_for "upstream_addr" $upstream_addr "proxy_protocol_addr" $proxy_protocol_addr "proxy_protocol_server_addr" $proxy_protocol_server_addr "request_uri" $request_uri "request" $request "http_x_forwarded_for" $http_x_forwarded_for "status" $status "args" $args';

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /dev/stdout devlogging;

        ##
        # Gzip Settings
        ##

        gzip on;

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                             html htm shtml;
    text/css                              css;
    text/xml                              xml;
    image/gif                             gif;
    image/jpeg                            jpeg jpg;
    application/javascript                js;
    application/atom+xml                  atom;
    application/rss+xml                   rss;

    text/mathml                           mml;
    text/plain                            txt;
    text/vnd.sun.j2me.app-descriptor      jad;
    text/vnd.wap.wml                      wml;
    text/x-component                      htc;

    image/avif                            avif;
    image/png                             png;
    image/svg+xml                         svg svgz;
    image/tiff                            tif tiff;
    image/vnd.wap.wbmp                    wbmp;
    image/webp                            webp;
    image/x-icon                          ico;
    image/x-jng                           jng;
    image/x-ms-bmp                        bmp;

    font/woff                             woff;
    font/woff2                            woff2;

    application/java-archive              jar war ear;
    application/json                      json;
    application/mac-binhex40              hqx;
    application/msword                    doc;
    application/pdf                       pdf;
    application/postscript                ps eps ai;
    application/rtf                       rtf;
    application/vnd.apple.mpegurl         m3u8;
    application/vnd.google-earth.kml+xml  kml;
    application/vnd.google-earth.kmz      kmz;
    application/vnd.ms-excel              xls;
    application/vnd.ms-fontobject         eot;
    application/vnd.ms-powerpoint         ppt;
    application/vnd.oasis.opendocument.graphics        odg;
    application/vnd.oasis.opendocument.presentation    odp;
    application/vnd.oasis.opendocument.spreadsheet     ods;
    application/vnd.oasis.opendocument.text            odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation    pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet    xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document    docx;
    application/vnd.wap.wmlc              wmlc;
    application/wasm                      wasm;
    application/x-7z-compressed           7z;
    application/x-cocoa                   cco;
    application/x-java-archive-diff       jardiff;
    application/x-java-jnlp-file          jnlp;
    application/x-makeself                run;
    application/x-perl                    pl pm;
    application/x-pilot                   prc pdb;
    application/x-rar-compressed          rar;
    application/x-redhat-package-manager  rpm;
    application/x-sea                     sea;
    application/x-shockwave-flash         swf;
    application/x-stuffit                 sit;
    application/x-tcl                     tcl tk;
    application/x-x509-ca-cert            der pem crt;
    application/x-xpinstall               xpi;
    application/xhtml+xml                 xhtml;
    application/xspf+xml                  xspf;
    application/zip                       zip;

    application/octet-stream              bin exe dll;
    application/octet-stream              deb;
    application/octet-stream              dmg;
    application/octet-stream              iso img;
    application/octet-stream              msi msp msm;

    audio/midi                            mid midi kar;
    audio/mpeg                            mp3;
    audio/ogg                             ogg;
    audio/x-m4a                           m4a;
    audio/x-realaudio                     ra;

    video/3gpp                            3gpp 3gp;
    video/mp2t                            ts;
    video/mp4                             mp4;
    video/mpeg                            mpeg mpg;
    video/quicktime                       mov;
    video/webm                            webm;
    video/x-flv                           flv;
    video/x-m4v                           m4v;
    video/x-mng                           mng;
    video/x-ms-asf                        asx asf;
    video/x-ms-wmv                        wmv;
    video/x-msvideo                       avi;
}

# configuration file /etc/nginx/sites-enabled/default:
server {
    listen       80 default_server;
    listen       [::]:80 default_server;
    server_name  _;
    root         /usr/share/nginx/html;
    proxy_http_version 1.1;
    client_max_body_size 100M;
    proxy_buffering off;
    proxy_set_header Connection "";

    # compression
    gzip            on;
    gzip_comp_level 1;
    gzip_http_version 1.0;
    gzip_min_length 512;
    gzip_proxied    expired no-cache no-store private auth;
    gzip_types  text/plain
                text/css
                application/json
                application/javascript
                application/x-javascript
                text/xml application/xml
                application/xml+rss
                text/javascript
                application/x-font-ttf
                font/woff2
                image/svg+xml
                image/x-icon;

    # Load configuration files for the default server block.
    include /etc/nginx/default.d/*.conf;

    location / {
        try_files $uri $uri/ /index.html;
    }

    location /version.json {
        add_header Cache-Control 'no-cache';
    }

    location /api {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $proxy_host;
        proxy_pass 
;
        rewrite /api/(.*) /$1  break;
    }

    location /widgets {
        alias /usr/share/nginx/widgets;
        try_files $uri $uri/ /widgets/index.html;
        add_header Content-Security-Policy "frame-ancestors *";
    }


    error_page 404 /404.html;
        location = /40x.html {
    }

    error_page 500 502 503 504 /50x.html;
        location = /50x.html {
    }
}

(same issue btw)

  
  
Posted one month ago

one thing I can think of is nginx default client_max_body_size
limit
None
800KB is about 1MiB
can you try altering nginx configuration in the running web instance and reload the new config?
nginx -s reload

  
  
Posted one month ago