Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
We Have This Issue With Our Oss Deployment: One Azure Container App For

We have this issue with our OSS deployment: one Azure Container App for web server and one Azure Container App for api server .
The issue: the console log don't display in the WebUI for some of our task (~800kb)
Digging further we found that the console result is corrupted. We extracted the curl command:

curl '
'   --compressed   -X POST   -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0'   -H 'Accept: application/json'   -H 'Accept-Language: en-US,en;q=0.5'   -H 'Accept-Encoding: gzip, deflate, br, zstd'   -H 'X-Allegro-Client: Webapp-2.2.0-690'   -H 'Content-Type: application/json'   -H 'Origin: 
'   -H 'DNT: 1'   -H 'Connection: keep-alive'   -H 'Referer: 
..azurecontainerapps.io/projects/*/tasks/10ba913ad81642faaf5d0e3ff1c01aeb/log?columns=selected^&columns=type^&columns=name^&columns=last_iteration^&columns=parent.name^&order=-last_update^&filter='   -H 'Cookie: clearml_token_basic=REDACTED'   -H 'Sec-Fetch-Dest: empty'   -H 'Sec-Fetch-Mode: cors'   -H 'Sec-Fetch-Site: same-origin'   -H 'Sec-GPC: 1'   -H 'TE: trailers'   --data-raw '{"task":"10ba913ad81642faaf5d0e3ff1c01aeb","batch_size":1000,"navigate_earlier":true}

When running this curl: 8/10 we get curl: (18) HTTP/2 stream 1 was reset With about 56009 received on total of 87025

The log on the web container show:

[error] 13#13: *41888 upstream prematurely closed connection while reading upstream, client: 100.100.1.217, server: _, request: "POST /api/v2.33/events.get_task_log HTTP/1.1", upstream: "
", host: "REDACTED-web.azurecontainerapps.io", referrer: "
"

We though first that the issue was between the api server and our ES server, as suggested here .
But then I tried to query directly the api server but slightly modifying the address:

curl '
'  ....

And I get the full json 100% of the time.

Which means: there are some issue in the communication between api and web
They are 2 Azure Container App running in the same Azure Container App Environment. The log never show any sign of issue between the two a part from this task log.

May be it's an issue in how the web forward data from the api to the client ?? @<1523701070390366208:profile|CostlyOstrich36>

  
  
Posted 2 months ago
Votes Newest

Answers 16


can you make sure the change to client_max_body_size was applied with nginx -T ?

  
  
Posted 2 months ago

@<1523703436166565888:profile|DeterminedCrab71> Found the solution:
We needed proxy_set_header Connection ""; in the location /api section. Putting it above in the server section had no effect

  
  
Posted 2 months ago

one thing I can think of is nginx default client_max_body_size
limit
None
800KB is about 1MiB
can you try altering nginx configuration in the running web instance and reload the new config?
nginx -s reload

  
  
Posted 2 months ago

I did.
I am now redeploying to new container to be sure.

  
  
Posted 2 months ago

Should I raise a github issue ticket ?

  
  
Posted 2 months ago

None

  
  
Posted 2 months ago

yes thanks, that would be very helpful

  
  
Posted 2 months ago

yes, I did try that exactly: all curl request succeed

  
  
Posted 2 months ago

can you try running the curl request from within the web container?

  
  
Posted 2 months ago

on second thought if direct request to api server works while reverse proxy fails, this points us only to nginx.

  
  
Posted 2 months ago

Deployed fresh new and ran nginx -T in the container:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
error_log stderr notice;

events {
        worker_connections 768;
        # multi_accept on;
}

http {
        client_max_body_size 100M;
        rewrite_log on;

        log_format devlogging '$remote_addr - $remote_user [$time_local] "proxy_host" $proxy_host "proxy_add_x_forwarded_for" $proxy_add_x_forwarded_for "upstream_addr" $upstream_addr "proxy_protocol_addr" $proxy_protocol_addr "proxy_protocol_server_addr" $proxy_protocol_server_addr "request_uri" $request_uri "request" $request "http_x_forwarded_for" $http_x_forwarded_for "status" $status "args" $args';

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /dev/stdout devlogging;

        ##
        # Gzip Settings
        ##

        gzip on;

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                             html htm shtml;
    text/css                              css;
    text/xml                              xml;
    image/gif                             gif;
    image/jpeg                            jpeg jpg;
    application/javascript                js;
    application/atom+xml                  atom;
    application/rss+xml                   rss;

    text/mathml                           mml;
    text/plain                            txt;
    text/vnd.sun.j2me.app-descriptor      jad;
    text/vnd.wap.wml                      wml;
    text/x-component                      htc;

    image/avif                            avif;
    image/png                             png;
    image/svg+xml                         svg svgz;
    image/tiff                            tif tiff;
    image/vnd.wap.wbmp                    wbmp;
    image/webp                            webp;
    image/x-icon                          ico;
    image/x-jng                           jng;
    image/x-ms-bmp                        bmp;

    font/woff                             woff;
    font/woff2                            woff2;

    application/java-archive              jar war ear;
    application/json                      json;
    application/mac-binhex40              hqx;
    application/msword                    doc;
    application/pdf                       pdf;
    application/postscript                ps eps ai;
    application/rtf                       rtf;
    application/vnd.apple.mpegurl         m3u8;
    application/vnd.google-earth.kml+xml  kml;
    application/vnd.google-earth.kmz      kmz;
    application/vnd.ms-excel              xls;
    application/vnd.ms-fontobject         eot;
    application/vnd.ms-powerpoint         ppt;
    application/vnd.oasis.opendocument.graphics        odg;
    application/vnd.oasis.opendocument.presentation    odp;
    application/vnd.oasis.opendocument.spreadsheet     ods;
    application/vnd.oasis.opendocument.text            odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation    pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet    xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document    docx;
    application/vnd.wap.wmlc              wmlc;
    application/wasm                      wasm;
    application/x-7z-compressed           7z;
    application/x-cocoa                   cco;
    application/x-java-archive-diff       jardiff;
    application/x-java-jnlp-file          jnlp;
    application/x-makeself                run;
    application/x-perl                    pl pm;
    application/x-pilot                   prc pdb;
    application/x-rar-compressed          rar;
    application/x-redhat-package-manager  rpm;
    application/x-sea                     sea;
    application/x-shockwave-flash         swf;
    application/x-stuffit                 sit;
    application/x-tcl                     tcl tk;
    application/x-x509-ca-cert            der pem crt;
    application/x-xpinstall               xpi;
    application/xhtml+xml                 xhtml;
    application/xspf+xml                  xspf;
    application/zip                       zip;

    application/octet-stream              bin exe dll;
    application/octet-stream              deb;
    application/octet-stream              dmg;
    application/octet-stream              iso img;
    application/octet-stream              msi msp msm;

    audio/midi                            mid midi kar;
    audio/mpeg                            mp3;
    audio/ogg                             ogg;
    audio/x-m4a                           m4a;
    audio/x-realaudio                     ra;

    video/3gpp                            3gpp 3gp;
    video/mp2t                            ts;
    video/mp4                             mp4;
    video/mpeg                            mpeg mpg;
    video/quicktime                       mov;
    video/webm                            webm;
    video/x-flv                           flv;
    video/x-m4v                           m4v;
    video/x-mng                           mng;
    video/x-ms-asf                        asx asf;
    video/x-ms-wmv                        wmv;
    video/x-msvideo                       avi;
}

# configuration file /etc/nginx/sites-enabled/default:
server {
    listen       80 default_server;
    listen       [::]:80 default_server;
    server_name  _;
    root         /usr/share/nginx/html;
    proxy_http_version 1.1;
    client_max_body_size 100M;
    proxy_buffering off;
    proxy_set_header Connection "";

    # compression
    gzip            on;
    gzip_comp_level 1;
    gzip_http_version 1.0;
    gzip_min_length 512;
    gzip_proxied    expired no-cache no-store private auth;
    gzip_types  text/plain
                text/css
                application/json
                application/javascript
                application/x-javascript
                text/xml application/xml
                application/xml+rss
                text/javascript
                application/x-font-ttf
                font/woff2
                image/svg+xml
                image/x-icon;

    # Load configuration files for the default server block.
    include /etc/nginx/default.d/*.conf;

    location / {
        try_files $uri $uri/ /index.html;
    }

    location /version.json {
        add_header Cache-Control 'no-cache';
    }

    location /api {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $proxy_host;
        proxy_pass 
;
        rewrite /api/(.*) /$1  break;
    }

    location /widgets {
        alias /usr/share/nginx/widgets;
        try_files $uri $uri/ /widgets/index.html;
        add_header Content-Security-Policy "frame-ancestors *";
    }


    error_page 404 /404.html;
        location = /40x.html {
    }

    error_page 500 502 503 504 /50x.html;
        location = /50x.html {
    }
}

(same issue btw)

  
  
Posted 2 months ago

will send the nginx -T results once the container is deployed

  
  
Posted 2 months ago

i'd focus on nginx or whatever azure infra you have between nginx and the client.

  
  
Posted 2 months ago

@<1523703436166565888:profile|DeterminedCrab71> Thansk for the suggestion. But no effect.
We already have client_max_body_size 0; in the server section
I tried to set both http and server section 100M but nothing changes.

Do you think the gzip be related ?

  
  
Posted 2 months ago

and I tried the curl against the web url: and it fails 8/10

  
  
Posted 2 months ago

the weird things is if it's a Azure ACA issue, it would be known right ? There are so many people who use ACA and having ACA talking to each other.
this is really weird ...

  
  
Posted 2 months ago