Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, I'M Using Fractional Gpu Container, But I Can'T Profile Any Thing, The Nsys Says Only "Segmentation Fault (Core Dumped)" Can You Guys Help Me And Do You Know What Is Wrong With This?

Hi, I'm using fractional gpu container, but I can't profile any thing, the nsys says only "Segmentation fault (core dumped)"

Can you guys help me and do you know what is wrong with this?

  
  
Posted 21 days ago
Votes Newest

Answers 9


Hi, can you provide the full log?

  
  
Posted 21 days ago

Thanks for your reply.

Environment of host system:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.29.06              Driver Version: 545.29.06    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090        Off | 00000000:01:00.0 Off |                  N/A |
| 33%   52C    P0             115W / 350W |      4MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        Off | 00000000:03:00.0 Off |                  N/A |
| 30%   49C    P0             108W / 350W |      4MiB / 24576MiB |      5%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.6 LTS
Release:        18.04
Codename:       bionic

I installed cuda-toolkit-12.3 following this url on fractional gpu container. After the setup, I ran

nsys profile nvidia-smi (or other cuda program)

It says:

Segmentation fault (core dumped)

Tell me if you need another information. Thanks again.

  
  
Posted 21 days ago

What command did you run? What were you trying to do? What was the setup?

  
  
Posted 20 days ago

Wait a minute...

  
  
Posted 20 days ago

  • Install the docker container which clearml provides (clearml/fractional-gpu:u22-cu12.3-4gb)
  • Run
docker run -it --gpus all --ipc=host --pid=host clearml/fractional-gpu:u22-cu12.3-8gb bash
  1. Install the cuda toolkit
wget 

mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget 

dpkg -i cuda-repo-ubuntu2204-12-3-local_12.3.0-545.23.06-1_amd64.deb
cp /var/cuda-repo-ubuntu2204-12-3-local/cuda-*-keyring.gpg /usr/share/keyrings/
apt-get update
apt-get -y install cuda-toolkit-12-3
  1. Run
nsys profile nvidia-smi
  
  
Posted 20 days ago

Addtionally, my container was clearml/fractional-gpu:u22-cu12.3-4gb (other containers based on cu12.3 also show that error too).

  
  
Posted 21 days ago

Can you provide the log though? Where you got there error?

  
  
Posted 20 days ago

That "Segmentation fault (core dumped)' was all I got.

  
  
Posted 20 days ago

I ran nsys profiler inside of the container.

  
  
Posted 20 days ago
137 Views
9 Answers
21 days ago
20 days ago
Tags