WickedGoat98

13 Questions, 96 Answers

Active since 10 January 2023

Last activity 25 days ago

Reputation

Badges 1

96 × Eureka!

Questions 13
Answers 96

0 Votes

0 Answers

601 Views

0 Votes 0 Answers 601 Views

Also The First Run, Even If It Shows Me Its Output And Is Done Is Marked Still As Running

also the first run, even if it shows me its output and is done is marked still as running 😞

clearml

3 years ago

0 Votes

4 Answers

578 Views

0 Votes 4 Answers 578 Views

Hey Guys, I Just Published (My Second Over All) Article On Medium

Hey guys, I just published (my second over all) article on Medium 😁 https://medium.com/@w.goutas/track-your-experiments-with-clearml-b26b2d0d6518

clearml

3 years ago

0 Votes

18 Answers

580 Views

0 Votes 18 Answers 580 Views

Hi All, I'M Trying To Deploy Trains On Rancher (Nice Kubernetes Cluster Orchestration Project) Where I'M Quite New To Rancher And Kubernetes. I Have Been Able To Install Trains Using Helm

Hi all, I'm trying to deploy trains on Rancher (nice Kubernetes cluster orchestration project) where I'm quite new to Rancher and Kubernetes. I have been abl...

clearml

3 years ago

0 Votes

28 Answers

643 Views

0 Votes 28 Answers 643 Views

Hi Guys, I Managed To Set Up A Kubernetes Cluster And Install Trains Into It. While Testing My Set-Up I Run The Test_Reporting.Py Example

Hi guys, I managed to set up a Kubernetes cluster and install Trains into it. While testing my set-up I run the test_reporting.py example https://github.com/...

clearml

3 years ago

0 Votes

5 Answers

528 Views

0 Votes 5 Answers 528 Views

Hi, I Have A Pre-Processing Steps Not Been Implemented In Python, But Being A Shell Script Calling Wget To Synchronize Data And Creating Intermediate Sqlite Dbs By A Script Been Implemented In 'R' And Would Like To Ask, If Trains Can Be Used Just To Trigg

Hi, I have a pre-processing steps not been implemented in Python, but being a shell script calling wget to synchronize data and creating intermediate SQLite ...

clearml

3 years ago

0 Votes

5 Answers

674 Views

0 Votes 5 Answers 674 Views

Hello Guys, I Read About Trains Some Days Ago And Think It Is Exectly What I Was Looking For, So I Ran The Docker Image And Started Thinking Of What I Would Like To Do And The Processing Steps I Would Like To Automize Which I Currently Run Manually Trigge

Hello guys, I read about Trains some days ago and think it is exectly what I was looking for, so I ran the docker image and started thinking of what I would ...

kubernetes

3 years ago

0 Votes

0 Answers

576 Views

0 Votes 0 Answers 576 Views

Hi Guys, My Second Article About Clearml Is Out

Hi guys, my second article about ClearML is out 🙂 https://w-goutas.medium.com/track-your-experiments-with-clearml-in-docker-37c5bdb3c582 thanks for reading 😉

clearml

3 years ago

0 Votes

17 Answers

652 Views

0 Votes 17 Answers 652 Views

Hi, I Run The Trains Server In An Docker Container And Started Making Use Of Tasks ... My Tests Are Showed On The Projects Dashboard Which Is Realy Cool. What I Haven'T Found So Far Is A Way To Clean Up The System From The Tests I Did. I'M Able To Archive

Hi, I run the trains server in an docker container and started making use of tasks ... My tests are showed on the Projects dashboard which is realy cool. Wha...

clearml

3 years ago

0 Votes

23 Answers

610 Views

0 Votes 23 Answers 610 Views

Hi, I Need Your Help Setting Up An Trains Agent Running In Docker. I Have An Python Script Calling Wget As System Command Which Runs Fine On My Dev Engine. When Cloning The Experiment And Scheduling It Into The Services Queue I Get An Error That The Call

Hi, I need your help setting up an trains agent running in docker. I have an python script calling wget as system command which runs fine on my dev engine. W...

mlops

3 years ago

0 Votes

26 Answers

560 Views

0 Votes 26 Answers 560 Views

Hi, I Try To Write An Article On Medium About Clearml And Face Some A Problem With Plotly Figures. When Displaying The Figure Locally In A Browser Works Fine, But On The Cleaml Server (I Use The Free Tier Service) The Plot Is Empty And Has The Title 'Unkn

Hi, I try to write an article on medium about ClearML and face some a problem with plotly figures. When displaying the figure locally in a browser works fine...

clearml

3 years ago

0 Votes

4 Answers

579 Views

0 Votes 4 Answers 579 Views

Hi, I'M Using The Dockerized Version Of Trains Get An Understanding Of Trains. While Trying To Play With The Trains.Conf Settings In ~/Trains.Conf I Got In A State, Where The Agent Is Not Been Able To Clone My Repo From

Hi, I'm using the dockerized version of trains get an understanding of trains. While trying to play with the trains.conf settings in ~/trains.conf I got in a...

mlops

3 years ago

0 Votes

5 Answers

676 Views

0 Votes 5 Answers 676 Views

I Cloned It And Scheduled It To The Default Queue, But It Is Not Being Processed. Is The Default Queue By Default Not Usable?

I cloned it and scheduled it to the default queue, but it is not being processed. Is the default queue by default not usable?

clearml

3 years ago

0 Votes

25 Answers

597 Views

0 Votes 25 Answers 597 Views

Hi, I Expect There Is A Limitation In Time The Free Service

Hi, I expect there is a limitation in time the free service http://app.community.clear.ml is storing projects and experiments of an user. Can you please tell...

clearml

3 years ago

0 Hi Guys, I Managed To Set Up A Kubernetes Cluster And Install Trains Into It. While Testing My Set-Up I Run The Test_Reporting.Py Example

I'm quite new to Kubernetes. What I have found is that the ports I expected, are used
` root@vmd62521:~# kubectl get services -n trains
NAME                   TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)         AGE
mongo-service          ClusterIP  10.43.99.44    <none>       27017/TCP       25h
webserver-service      NodePort   10.43.49.21    <none>       80:30080/TCP    25h
redis                  ClusterIP  10.43.62.222   <none>       6379/TCP        25h
elasticsearch-service  Clust...

3 years ago

0 Hi Guys, I Managed To Set Up A Kubernetes Cluster And Install Trains Into It. While Testing My Set-Up I Run The Test_Reporting.Py Example

the ports, I'm not sure about

3 years ago

0 Hi Guys, I Managed To Set Up A Kubernetes Cluster And Install Trains Into It. While Testing My Set-Up I Run The Test_Reporting.Py Example

I will check

3 years ago

0 Hi Guys, I Managed To Set Up A Kubernetes Cluster And Install Trains Into It. While Testing My Set-Up I Run The Test_Reporting.Py Example

api_server and web_server look ok
(py38) wgo@NVidia-power:~/dev/Trains/trains$ curl {"meta":{"id":"bb5cd73435fb4127b9509ce3a771e95b","trx":"bb5cd73435fb4127b9509ce3a771e95b","endpoint":{"name":"","requested_version":1.0,"actual_version":null},"result_code":400,"result_spath /","error_stack":null},"data":{}}(py38) wgo@NVidia-power:~/dev/Trains/trains$ curl `
<!doctype html>

<head> <meta charset="utf-8"> <title>trains</title> <base href="/"> <meta name="vie...

3 years ago

0 Hi Guys, I Managed To Set Up A Kubernetes Cluster And Install Trains Into It. While Testing My Set-Up I Run The Test_Reporting.Py Example

also the webserver pods log contains entries

3 years ago

0 Hi Guys, I Managed To Set Up A Kubernetes Cluster And Install Trains Into It. While Testing My Set-Up I Run The Test_Reporting.Py Example

redis, mongo and elasticsearch looks also ok

3 years ago

0 Hi Guys, I Managed To Set Up A Kubernetes Cluster And Install Trains Into It. While Testing My Set-Up I Run The Test_Reporting.Py Example

the apiserver pods reports quite a lot

3 years ago

0 Hi Guys, I Managed To Set Up A Kubernetes Cluster And Install Trains Into It. While Testing My Set-Up I Run The Test_Reporting.Py Example

AgitatedDove14 I don't know why, but now it works
runfile('/home/wgo/dev/Trains/trains/examples/reporting/text_reporting.py', wdir='/home/wgo/dev/Trains/trains/examples/reporting') TRAINS Task: overwriting (reusing) task id=b31459aa2d414ea7b5aaa8c467ee6ad3 This is standard error test 2020-12-12 11:51:44.841 | INFO | __main__:report_logs:26 - That's it, beautiful and simple logging! (using ANSI colors) TRAINS results page: `
reporting text logs
This is standard output test
hello, th...

3 years ago

0 Hi, I Need Your Help Setting Up An Trains Agent Running In Docker. I Have An Python Script Calling Wget As System Command Which Runs Fine On My Dev Engine. When Cloning The Experiment And Scheduling It Into The Services Queue I Get An Error That The Call

AgitatedDove14 regarding the credentials, will I need to take them out of my trains.conf, or might it be common practise to create a user for such pods instantiating additional workers listening on queues?

3 years ago

I have been able to make use of
image: allegroai/trains-agent:latest
in the docker-compose file.yml 🎉
now I will focus on getting it working on Rancher
stay tuned

3 years ago

👍

3 years ago

0 Hi, I Have A Pre-Processing Steps Not Been Implemented In Python, But Being A Shell Script Calling Wget To Synchronize Data And Creating Intermediate Sqlite Dbs By A Script Been Implemented In 'R' And Would Like To Ask, If Trains Can Be Used Just To Trigg

but before I need to understand how parameters are processed. See my last question in my earlier https://app.slack.com/client/TT9ATQXJ5/CTK20V944/thread/CTK20V944-1603740766.425000

3 years ago

0 Hi, I'M Using The Dockerized Version Of Trains Get An Understanding Of Trains. While Trying To Play With The Trains.Conf Settings In ~/Trains.Conf I Got In A State, Where The Agent Is Not Been Able To Clone My Repo From

AgitatedDove14 I use docker

3 years ago

Ok, thanks for the support

3 years ago

I think I understand now, that the trains.conf has to be located on the node running the trains-agent.
When starting an additional trains-agent not been instantiated by docker-compose so it is not part of the same network, I get problems finding the api_server. localhost:8008 for sure will not be. I dentified the IP of the server running in docker with docker inspect ... and edited ~/trains.conf using it, but unfortunately it still cannot find the apiserver 😞
` (py38) wgo@NVidi...

3 years ago

AgitatedDove14 today I managed to run what I couldn't a month before:)
I didn't understand correctly what you wrote me that time.
The issue I had was, that I missed wget in the trains-agent image and was not able to run a system call of wget.
Now I mannaged to do so based on your imput you gave me by adding the
agent.docker_preprocess_bash_script = [...]in my trains.config, and it worked out of the box 🙂
Basically this issue was the reason why I started learning how to create a Kube...

3 years ago

AgitatedDove14 not sure how to make use of such config / where to add it
Is it to be added in the docker image when generating an own, or can I set this in the Web GUI as property of the experiment I cloned, shall it be added in the original script but type what kind of variable type is 'agent' of?

3 years ago

Thanks, will try on weekend to update the trains.conf

3 years ago

0 Hi, I Run The Trains Server In An Docker Container And Started Making Use Of Tasks ... My Tests Are Showed On The Projects Dashboard Which Is Realy Cool. What I Haven'T Found So Far Is A Way To Clean Up The System From The Tests I Did. I'M Able To Archive

regarding the clean-up servide, do I need to run this as cron job, or does the trains server support a kind of add-ons where I need to copy the script to?

3 years ago

another question I have is, are the models been trained stored (I guess they are stored) in the mongodb or in the file system and which format is been used ?

3 years ago

ok thanks, will need to run some tests later

3 years ago

I ran an local (not dockerized) trains-agent
trains-agent daemon --queue training --create-queue --foregroundwhich enabled me to see the GPU load on the corresponding view 🙂

Now I got another issue.
It seems when cloning an experiment, a virtual environment is been created with all the modules been identified to be used. Inside this environment the experiment is running.
Am I right?
Is this the case only for clones?

In my Python code I'm trying to read a pandas table which I stored i...

3 years ago

Sorry, but I don'T understand how the cloned experiment is been provided with parameters.
A task which is been cloned by Trains might get its parameter via task.set_parameters(dict)
this parameters are comming from soe magic analysis of the argparse been used in the script.
AgitatedDove14 when is the call to set_parameter(...) been performed? Is the argparse call been somehow redirected and will receive the data from Trains instead of getting them via sys.argv or wherever argparse is gettin...

3 years ago

after adding the
import fastparquet
statement to the code, the reconstruction of an clone is working
` Summary - installed python packages:
...

fastparquet==0.4.1
...
Environment setup completed successfully
Starting Task Execution:
...
modeller.py: error: the following arguments are required: --algorithm `unfortunately it raises the next issue.
If the script been used expects to get parameters via command line (which in Trains experiments are identified and stored as parameter when using...

3 years ago

well I managed to clone an experiment and adat its parameter on the trains server via browser.
If argparse is been used, no parameter must be defined as required. Instead it has to be managed by the script after parsing the parameter and something mandatory is missing to terminate.
Doing so worked fine for me 😁 at least for this part of work. Now fastparquet and missing packages are failing again...

3 years ago

thanks Martin

3 years ago

0 Hi Guys, I Managed To Set Up A Kubernetes Cluster And Install Trains Into It. While Testing My Set-Up I Run The Test_Reporting.Py Example

the log of the fileserver pod seems quite empty
` root@vmd62521:~# kubectl logs fileserver-6f49b74556-2m4n2 -n trains --all-containers

Serving Flask app "fileserver" (lazy loading)
Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
Debug mode: off
root@vmd62521:~# same to the agentservice root@vmd62521:~# kubectl logs agentservices-56655788b6-rnbk4 apiserver-7d9cd59844-dfd5s -n train...

3 years ago

AgitatedDove14 I still do not understand, how I can deploy the trains-agent docker image to my trains-server installation so the 'default' queue will be handled.
Once I can do this, it should not be a big thing to add additional workers for more queues.
I found a template for k8s but as I'm quite new to Kubernetes I don't know how to use it.
As I use Rancher I'm able to even edit the trains-agent deployment. I added an additional command to handle the default queue as well, but it seems not ...

3 years ago

Thanks a lot. I will let you know if I manged it :)

3 years ago

AgitatedDove14 ok, but how to deploy a trains-agent?

3 years ago

Show more results