How to create a compute instance & compute cluster using azure cli in ML workspace from local Linux machine

How to create a compute instance & compute cluster using azure cli in ML workspace from local linux machine

1) Step verify if there are any compute instance already alive

az resource list

>> pre value collected

az ml compute list -g "rg-dp100-labs" -w "mlw-dp100-labs"

2) Step create compute instance

compute intance:

az ml compute create --name "mlw-dp100-labs-compute01" --size Standard_A1_v2 --type ComputeInstance -w mlw-dp100-labs -g rg-dp100-labs

Actual output:

(azcli) [vagrant@localhost ~]$ az ml compute create --name "mlw-dp100-labs-compute01" --size Standard_A1_v2 --type ComputeInstance -w mlw-dp100-labs -g rg-dp100-labs
{
  "enable_node_public_ip": true,
  "id": "/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs/providers/Microsoft.MachineLearningServices/workspaces/mlw-dp100-labs/computes/mlw-dp100-labs-compute01",
  "last_operation": {
    "operation_name": "Create",
    "operation_status": "Succeeded",
    "operation_time": "2023-09-09T16:31:17.712Z",
    "operation_trigger": "User"
  },
  "location": "ukwest",
  "name": "mlw-dp100-labs-compute01",
  "network_settings": {
    "private_ip_address": "10.0.0.5",
    "public_ip_address": "51.141.124.49"
  },
  "os_image_metadata": {
    "current_image_version": "23.06.30",
    "is_latest_os_image_version": true,
    "latest_image_version": "23.06.30"
  },
  "provisioning_state": "Succeeded",
  "resourceGroup": "rg-dp100-labs",
  "services": [
    {
      "display_name": "Jupyter",
      "endpoint_uri": "https://mlw-dp100-labs-compute01.ukwest.instances.azureml.ms/tree/"
    },
    {
      "display_name": "Jupyter Lab",
      "endpoint_uri": "https://mlw-dp100-labs-compute01.ukwest.instances.azureml.ms/lab"
    }
  ],
  "size": "STANDARD_A1_V2",
  "ssh_public_access_enabled": false,
  "ssh_settings": {
    "admin_username": "azureuser",
    "ssh_port": "4001"
  },
  "state": "Running",
  "type": "computeinstance"
}

compute cluster:

az ml compute create --name "aml-cluster" --size Standard_A1_v2 --max-instances 2 --type AmlCompute -w mlw-dp100-labs -g rg-dp100-labs

Actual output:

(azcli) [vagrant@localhost ~]$ az ml compute create --name "aml-cluster" --size Standard_A1_v2 --max-instances 2 --type AmlCompute -w mlw-dp100-labs -g rg-dp100-labs

{
  "enable_node_public_ip": true,
  "id": "/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs/providers/Microsoft.MachineLearningServices/workspaces/mlw-dp100-labs/computes/aml-cluster",
  "idle_time_before_scale_down": 120,
  "location": "ukwest",
  "max_instances": 2,
  "min_instances": 0,
  "name": "aml-cluster",
  "network_settings": {},
  "provisioning_state": "Succeeded",
  "resourceGroup": "rg-dp100-labs",
  "size": "STANDARD_A1_V2",
  "ssh_public_access_enabled": true,
  "tier": "dedicated",
  "type": "amlcompute"
}
(azcli) [vagrant@localhost ~]$

3) Verify again the resource list and compute instances

az resource list

>> no difference to the past output. Since the compute instance and cluster are associated to ML workspace.

az ml compute list -g "rg-dp100-labs" -w "mlw-dp100-labs"

Actual output:

(azcli) [vagrant@localhost ~]$ az ml compute list -g "rg-dp100-labs" -w "mlw-dp100-labs"
[
  {
    "enable_node_public_ip": true,
    "id": "/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs/providers/Microsoft.MachineLearningServices/workspaces/mlw-dp100-labs/computes/mlw-dp100-labs-compute01",
    "last_operation": {
      "operation_name": "Create",
      "operation_status": "Succeeded",
      "operation_time": "2023-09-09T16:31:17.712Z",
      "operation_trigger": "User"
    },
    "location": "ukwest",
    "name": "mlw-dp100-labs-compute01",
    "network_settings": {
      "private_ip_address": "10.0.0.5",
      "public_ip_address": "51.141.124.49"
    },
    "os_image_metadata": {
      "current_image_version": "23.06.30",
      "is_latest_os_image_version": true,
      "latest_image_version": "23.06.30"
    },
    "provisioning_state": "Succeeded",
    "resourceGroup": "rg-dp100-labs",
    "services": [
      {
        "display_name": "Jupyter",
        "endpoint_uri": "https://mlw-dp100-labs-compute01.ukwest.instances.azureml.ms/tree/"
      },
      {
        "display_name": "Jupyter Lab",
        "endpoint_uri": "https://mlw-dp100-labs-compute01.ukwest.instances.azureml.ms/lab"
      }
    ],
    "size": "STANDARD_A1_V2",
    "ssh_public_access_enabled": false,
    "ssh_settings": {
      "admin_username": "azureuser",
      "ssh_port": "4001"
    },
    "state": "Running",
    "type": "computeinstance"
  },
  {
    "enable_node_public_ip": true,
    "id": "/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs/providers/Microsoft.MachineLearningServices/workspaces/mlw-dp100-labs/computes/aml-cluster",
    "idle_time_before_scale_down": 120,
    "location": "ukwest",
    "max_instances": 2,
    "min_instances": 0,
    "name": "aml-cluster",
    "network_settings": {},
    "provisioning_state": "Succeeded",
    "resourceGroup": "rg-dp100-labs",
    "size": "STANDARD_A1_V2",
    "ssh_public_access_enabled": true,
    "tier": "dedicated",
    "type": "amlcompute"
  }
]
(azcli) [vagrant@localhost ~]$

How to train a model using Azure ML

How to train a model using Azure ML

1) Launching a terminal for the compute instance

click on compute instance and then "..." near jupyter notebook and launch terminal (if still not launched, use "+" symbol to launch the terminal)

2) installing azure-ai-ml from the terminal

pip uninstall azure-ai-ml

>>> by default none installed

pip install azure-ai-ml

Actual output:

>>> copy/paste wasnt possible (it finished fine with exception for blog and file-share with compatible warnings)

3) cloning azure-ml-labs git repo

git clone https://github.com/MicrosoftLearning/mslearn-azure-ml.git azure-ml-labs

4)  Verify we get new folders created

Users/your-user-name/azure-ml-labs folder has been created.

>> yes we have

5) Open the Labs/02/"Run training script.ipynb" notebook.

6) Verify that the notebook uses the Python 3.8 - AzureML kernel. Each kernel has its own image with its own set of packages pre-installed.

7) Run all cells in the notebook.

>>> here we needed to reauthenticate the visual studio, post reauthentication, we can continue running the cell. The job should run and complete.

8) Check the job progress

Queued -> Preparing -> Running -> finalizing -> Completed

Once job is complete, we can click on the job to see the model's accuracy.

How to create a resource group & ML workspace in Azure using azure cli (az) from local linux machine

How to create a resource group & ML workspace in Azure using azure cli (az) from local linux machine

1) Login to Azure

az login

>>> the command will ask you to authenticate the connection using a code. Finish it before next step.

2) let us list the subscriptions we have:

az account show

3) List down the azure regions:

az account list-locations

Our preferred location:
          "id": "/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/locations/ukwest",
          "name": "ukwest",
          "subscriptionId": null

4) Create the azure resource group:

az group create --name "rg-dp100-labs" --location "ukwest"

output:

(azcli) [vagrant@localhost ~]$ az group create --name "rg-dp100-labs" --location "ukwest"
{
  "id": "/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs",
  "location": "ukwest",
  "managedBy": null,
  "name": "rg-dp100-labs",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": "Microsoft.Resources/resourceGroups"
}
(azcli) [vagrant@localhost ~]$

5) List the available resource groups:

Command:

az group list

Output:

(azcli) [vagrant@localhost ~]$ az group list
[
  {
    "id": "/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs",
    "location": "ukwest",
    "managedBy": null,
    "name": "rg-dp100-labs",
    "properties": {
      "provisioningState": "Succeeded"
    },
    "tags": null,
    "type": "Microsoft.Resources/resourceGroups"
  }
]
(azcli) [vagrant@localhost ~]$

6) Create Azure ML workspace

az ml workspace create --name "mlw-dp100-labs" -g "rg-dp100-labs"

Actual output:

(azcli) [vagrant@localhost ~]$ az ml workspace create --name "mlw-dp100-labs" -g "rg-dp100-labs"
Class ManagedNetwork: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
The deployment request mlw-dp100-labs-6349534 was accepted. ARM deployment URI for reference:
https://portal.azure.com//#blade/HubsExtension/DeploymentDetailsBlade/overview/id/%2Fsubscriptions%2F67ed360b-95af-4a7f-b398-f459c8118cc1%2FresourceGroups%2Frg-dp100-labs%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fmlw-dp100-labs-6349534
Creating Log Analytics Workspace: (mlwdp100logalyti0d1ce113  ) ..  Done (16s)
Creating Application Insights: (mlwdp100insightsef9fc48f  )  Done (18s)
Creating Key Vault: (mlwdp100keyvault9db8214d  )  Done (24s)
Creating Storage Account: (mlwdp100storageb8dadeb6f  )   Done (28s)
Creating AzureML Workspace: (mlw-dp100-labs  ) ..  Done (18s)
Total time : 47s
Class ManagedNetworkSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class ManagedNetworkStatusSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
{
  "application_insights": "/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs/providers/Microsoft.insights/components/mlwdp100insightsef9fc48f",
  "description": "mlw-dp100-labs",
  "discovery_url": "https://ukwest.api.azureml.ms/discovery",
  "display_name": "mlw-dp100-labs",
  "enable_data_isolation": false,
  "hbi_workspace": false,
  "id": "/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs/providers/Microsoft.MachineLearningServices/workspaces/mlw-dp100-labs",
  "identity": {
    "principal_id": "64a46edf-9d6d-4f59-837a-d7d2eb5079c2",
    "tenant_id": "4e050b03-5c99-495a-8385-689d04c7baed",
    "type": "system_assigned"
  },
  "key_vault": "/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs/providers/Microsoft.Keyvault/vaults/mlwdp100keyvault9db8214d",
  "location": "ukwest",
  "managed_network": {
    "isolation_mode": "disabled",
    "outbound_rules": []
  },
  "mlflow_tracking_uri": "azureml://ukwest.api.azureml.ms/mlflow/v1.0/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs/providers/Microsoft.MachineLearningServices/workspaces/mlw-dp100-labs",
  "name": "mlw-dp100-labs",
  "public_network_access": "Enabled",
  "resourceGroup": "rg-dp100-labs",
  "resource_group": "rg-dp100-labs",
  "storage_account": "/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs/providers/Microsoft.Storage/storageAccounts/mlwdp100storageb8dadeb6f",
  "tags": {
    "createdByToolkit": "cli-v2-1.9.0"
  }
}
(azcli) [vagrant@localhost ~]$

7) Verify the azure ml worskspace existance

az ml workspace list

(azcli) [vagrant@localhost ~]$ az ml workspace list
Class ManagedNetworkSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class ManagedNetworkStatusSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
[
  {
    "application_insights": "/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs/providers/Microsoft.insights/components/mlwdp100insightsef9fc48f",
    "description": "mlw-dp100-labs",
    "discovery_url": "https://ukwest.api.azureml.ms/discovery",
    "display_name": "mlw-dp100-labs",
    "enable_data_isolation": false,
    "hbi_workspace": false,
    "id": "/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs/providers/Microsoft.MachineLearningServices/workspaces/mlw-dp100-labs",
    "identity": {
      "principal_id": "64a46edf-9d6d-4f59-837a-d7d2eb5079c2",
      "tenant_id": "4e050b03-5c99-495a-8385-689d04c7baed",
      "type": "system_assigned"
    },
    "key_vault": "/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs/providers/Microsoft.Keyvault/vaults/mlwdp100keyvault9db8214d",
    "location": "ukwest",
    "managed_network": {
      "isolation_mode": "disabled",
      "outbound_rules": []
    },
    "mlflow_tracking_uri": "azureml://ukwest.api.azureml.ms/mlflow/v1.0/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs/providers/Microsoft.MachineLearningServices/workspaces/mlw-dp100-labs",
    "name": "mlw-dp100-labs",
    "public_network_access": "Enabled",
    "resourceGroup": "rg-dp100-labs",
    "resource_group": "rg-dp100-labs",
    "storage_account": "/subscriptions/67ed360b-95af-4a7f-b398-f459c8118cc1/resourceGroups/rg-dp100-labs/providers/Microsoft.Storage/storageAccounts/mlwdp100storageb8dadeb6f",
    "tags": {
      "createdByToolkit": "cli-v2-1.9.0"
    }
  }
]
(azcli) [vagrant@localhost ~]$

Setup Azure CLI Machine Learning (ml) extension on local linux machine to manage & automate all azure tasks

Setup Azure CLI Machine Learning (ml) extension

(azcli) [vagrant@localhost ~]$ az extension remove -n azure-cli-ml
The extension azure-cli-ml is not installed. Please install the extension via `az extension add -n azure-cli-ml`.

(azcli) [vagrant@localhost ~]$ az extension remove -n ml
The extension ml is not installed. Please install the extension via `az extension add -n ml`.
(azcli) [vagrant@localhost ~]$

Before extension installation:

(azcli) [vagrant@localhost ~]$ ls -altr ~/.azure/cliextensions
total 4
drwxrwxr-x. 6 vagrant vagrant 4096 Sep  9 14:51 ..
drwxrwxr-x. 2 vagrant vagrant    6 Sep  9 14:53 .
(azcli) [vagrant@localhost ~]$

Adding Azure Extension:

az extension add -n ml -y

(azcli) [vagrant@localhost ~]$ az extension add -n ml -y
(azcli) [vagrant@localhost ~]$

(azcli) [vagrant@localhost ~]$ echo $?
0
(azcli) [vagrant@localhost ~]$

How does it look after extension is added:

ls -altr ~/.azure/cliextensions

(azcli) [vagrant@localhost ~]$ ls -altr ~/.azure/cliextensions
total 12
drwxrwxr-x.  6 vagrant vagrant 4096 Sep  9 14:51 ..
drwxrwxr-x.  3 vagrant vagrant   16 Sep  9 14:58 .
drwxrwxr-x. 98 vagrant vagrant 4096 Sep  9 14:58 ml
(azcli) [vagrant@localhost ~]$ 

(azcli) [vagrant@localhost ~]$ ls -altr ~/.azure/cliextensions/ml|wc -l
105
(azcli) [vagrant@localhost ~]$ 

(azcli) [vagrant@localhost ~]$ ls -altr ~/.azure/cliextensions/ml|more
total 8312
drwxrwxr-x.  2 vagrant vagrant     117 Sep  9 14:58 opencensus_context-0.1.3.dist-info
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 azure_common-1.1.28.dist-info
drwxrwxr-x.  4 vagrant vagrant    4096 Sep  9 14:58 websocket
drwxrwxr-x.  2 vagrant vagrant     126 Sep  9 14:58 websocket_client-1.6.2.dist-info
drwxrwxr-x.  6 vagrant vagrant    4096 Sep  9 14:58 urllib3
drwxrwxr-x.  2 vagrant vagrant     106 Sep  9 14:58 urllib3-1.26.16.dist-info
-rw-rw-r--.  1 vagrant vagrant  111082 Sep  9 14:58 typing_extensions.py
drwxrwxr-x.  2 vagrant vagrant      81 Sep  9 14:58 typing_extensions-4.7.1.dist-info
drwxrwxr-x.  4 vagrant vagrant    4096 Sep  9 14:58 tqdm
drwxrwxr-x.  2 vagrant vagrant     126 Sep  9 14:58 tqdm-4.66.1.dist-info
-rw-rw-r--.  1 vagrant vagrant   34549 Sep  9 14:58 six.py
drwxrwxr-x.  2 vagrant vagrant      74 Sep  9 14:58 __pycache__
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 six-1.16.0.dist-info
drwxrwxr-x.  3 vagrant vagrant     124 Sep  9 14:58 rpds
drwxrwxr-x.  3 vagrant vagrant      87 Sep  9 14:58 rpds_py-0.10.2.dist-info
drwxrwxr-x.  3 vagrant vagrant    4096 Sep  9 14:58 jwt
drwxrwxr-x.  2 vagrant vagrant     121 Sep  9 14:58 PyJWT-2.8.0.dist-info
drwxrwxr-x.  3 vagrant vagrant    4096 Sep  9 14:58 pydash
drwxrwxr-x.  2 vagrant vagrant     125 Sep  9 14:58 pydash-5.1.2.dist-info
drwxrwxr-x.  4 vagrant vagrant    4096 Sep  9 14:58 pycparser
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 pycparser-2.21.dist-info
drwxrwxr-x.  6 vagrant vagrant     115 Sep  9 14:58 pyasn1
drwxrwxr-x.  2 vagrant vagrant     122 Sep  9 14:58 pyasn1-0.5.0.dist-info
drwxrwxr-x.  4 vagrant vagrant    4096 Sep  9 14:58 psutil
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 psutil-5.9.5.dist-info
drwxrwxr-x.  2 vagrant vagrant      81 Sep  9 14:58 protobuf-4.24.3.dist-info
drwxrwxr-x.  3 vagrant vagrant     175 Sep  9 14:58 portalocker
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 portalocker-2.7.0.dist-info
drwxrwxr-x.  3 vagrant vagrant    4096 Sep  9 14:58 packaging
drwxrwxr-x.  2 vagrant vagrant     122 Sep  9 14:58 packaging-23.1.dist-info
drwxrwxr-x.  6 vagrant vagrant     144 Sep  9 14:58 oauthlib
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 oauthlib-3.2.2.dist-info
drwxrwxr-x.  3 vagrant vagrant     190 Sep  9 14:58 idna
drwxrwxr-x.  2 vagrant vagrant      84 Sep  9 14:58 idna-3.4.dist-info
drwxrwxr-x.  4 vagrant vagrant     149 Sep  9 14:58 colorama
drwxrwxr-x.  3 vagrant vagrant      82 Sep  9 14:58 colorama-0.4.6.dist-info
drwxrwxr-x.  5 vagrant vagrant    4096 Sep  9 14:58 charset_normalizer
drwxrwxr-x.  2 vagrant vagrant     126 Sep  9 14:58 charset_normalizer-3.2.0.dist-info
drwxrwxr-x.  3 vagrant vagrant     112 Sep  9 14:58 certifi
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 certifi-2023.7.22.dist-info
drwxrwxr-x.  3 vagrant vagrant      74 Sep  9 14:58 cachetools
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 cachetools-5.3.1.dist-info
drwxrwxr-x.  3 vagrant vagrant    4096 Sep  9 14:58 attr
drwxrwxr-x.  3 vagrant vagrant     179 Sep  9 14:58 attrs
drwxrwxr-x.  3 vagrant vagrant      82 Sep  9 14:58 attrs-23.1.0.dist-info
drwxrwxr-x.  3 vagrant vagrant    4096 Sep  9 14:58 rsa
drwxrwxr-x.  2 vagrant vagrant     105 Sep  9 14:58 rsa-4.9.dist-info
drwxrwxr-x.  3 vagrant vagrant    4096 Sep  9 14:58 requests
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 requests-2.31.0.dist-info
drwxrwxr-x.  4 vagrant vagrant    4096 Sep  9 14:58 referencing
drwxrwxr-x.  3 vagrant vagrant      82 Sep  9 14:58 referencing-0.30.2.dist-info
drwxrwxr-x.  6 vagrant vagrant    4096 Sep  9 14:58 dateutil
drwxrwxr-x.  2 vagrant vagrant     118 Sep  9 14:58 python_dateutil-2.8.2.dist-info
drwxrwxr-x.  3 vagrant vagrant    4096 Sep  9 14:58 pyasn1_modules
drwxrwxr-x.  2 vagrant vagrant     122 Sep  9 14:58 pyasn1_modules-0.3.0.dist-info
drwxrwxr-x.  3 vagrant vagrant    4096 Sep  9 14:58 marshmallow
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 marshmallow-3.20.1.dist-info
drwxrwxr-x.  4 vagrant vagrant    4096 Sep  9 14:58 isodate
drwxrwxr-x.  2 vagrant vagrant      87 Sep  9 14:58 isodate-0.6.1.dist-info
-rw-rw-r--.  1 vagrant vagrant    1710 Sep  9 14:58 googleapis_common_protos-1.60.0-py3.9-nspkg.pth
drwxrwxr-x.  2 vagrant vagrant     132 Sep  9 14:58 googleapis_common_protos-1.60.0.dist-info
-rwxrwxr-x.  1 vagrant vagrant 1065976 Sep  9 14:58 _cffi_backend.cpython-311-x86_64-linux-gnu.so
drwxrwxr-x.  3 vagrant vagrant    4096 Sep  9 14:58 cffi
drwxrwxr-x.  2 vagrant vagrant     126 Sep  9 14:58 cffi-1.15.1.dist-info
drwxrwxr-x.  4 vagrant vagrant    4096 Sep  9 14:58 strictyaml
drwxrwxr-x.  2 vagrant vagrant     106 Sep  9 14:58 strictyaml-1.7.3.dist-info
drwxrwxr-x.  4 vagrant vagrant     162 Sep  9 14:58 requests_oauthlib
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 requests_oauthlib-1.3.1.dist-info
drwxrwxr-x.  5 vagrant vagrant      88 Sep  9 14:58 jsonschema_specifications
drwxrwxr-x.  3 vagrant vagrant      82 Sep  9 14:58 jsonschema_specifications-2023.7.1.dist-info
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 google_auth-2.22.0.dist-info
drwxrwxr-x. 10 vagrant vagrant    4096 Sep  9 14:58 docker
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 docker-6.1.3.dist-info
drwxrwxr-x.  5 vagrant vagrant     160 Sep  9 14:58 cryptography
drwxrwxr-x.  2 vagrant vagrant     143 Sep  9 14:58 cryptography-41.0.3.dist-info
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 azure_core-1.29.4.dist-info
drwxrwxr-x.  6 vagrant vagrant    4096 Sep  9 14:58 msrest
drwxrwxr-x.  2 vagrant vagrant     105 Sep  9 14:58 msrest-0.7.1.dist-info
drwxrwxr-x.  5 vagrant vagrant    4096 Sep  9 14:58 jsonschema
drwxrwxr-x.  2 vagrant vagrant     190 Sep  9 14:58 bin
drwxrwxr-x.  3 vagrant vagrant     106 Sep  9 14:58 jsonschema-4.19.0.dist-info
-rw-rw-r--.  1 vagrant vagrant     539 Sep  9 14:58 google_api_core-2.11.1-py3.9-nspkg.pth
drwxrwxr-x. 14 vagrant vagrant     170 Sep  9 14:58 google
drwxrwxr-x.  2 vagrant vagrant     132 Sep  9 14:58 google_api_core-2.11.1.dist-info
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 azure_storage_file_share-12.13.0.dist-info
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 azure_storage_blob-12.17.0.dist-info
drwxrwxr-x.  2 vagrant vagrant     120 Sep  9 14:58 azure_mgmt_core-1.4.0.dist-info
drwxrwxr-x.  2 vagrant vagrant     132 Sep  9 14:58 opencensus-0.11.2.dist-info
drwxrwxr-x.  4 vagrant vagrant    4096 Sep  9 14:58 msal
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 msal-1.23.0.dist-info
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 azure_storage_file_datalake-12.12.0.dist-info
drwxrwxr-x.  2 vagrant vagrant      87 Sep  9 14:58 azure_mgmt_resourcegraph-8.0.0.dist-info
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 azure_mgmt_resource-22.0.0.dist-info
drwxrwxr-x.  3 vagrant vagrant     161 Sep  9 14:58 msal_extensions
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 msal_extensions-1.0.0.dist-info
drwxrwxr-x.  8 vagrant vagrant      91 Sep  9 14:58 azure
drwxrwxr-x.  2 vagrant vagrant     102 Sep  9 14:58 azure_identity-1.14.0.dist-info
drwxrwxr-x. 10 vagrant vagrant     133 Sep  9 14:58 opencensus
drwxrwxr-x.  2 vagrant vagrant     117 Sep  9 14:58 opencensus_ext_azure-1.1.9.dist-info
drwxrwxr-x.  6 vagrant vagrant     149 Sep  9 14:58 azext_mlv2
drwxrwxr-x.  2 vagrant vagrant     171 Sep  9 14:58 ml-2.19.1.dist-info
drwxrwxr-x.  3 vagrant vagrant      16 Sep  9 14:58 ..
drwxrwxr-x. 98 vagrant vagrant    4096 Sep  9 14:58 .
-rw-rw-r--.  1 vagrant vagrant 7175844 Sep  9 14:58 ml-2.19.1-py3-none-any.whl
(azcli) [vagrant@localhost ~]$

YouTube Video:

Installing Azure CLI in conda virtual environment

Note: we aren't using yum or dnf here, we are going to using virtualenv to manage the packages.

OS version: Oracle Linux 8.8

Option 1: curl based method using install.py script

https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=script

Result: Failed

Error:

(azcli) [vagrant@localhost azureinstall]$ curl -L https://aka.ms/InstallAzureCli | bash

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100  1405  100  1405    0     0   1703      0 --:--:-- --:--:-- --:--:--  4390

Downloading Azure CLI install script from https://azurecliprod.blob.core.windows.net/install.py to /tmp/azure_cli_install_tmp_zF6J3v.

######################################################################## 100.0%

...

Also creating executable in /home/vagrant/lib/azure-cli/bin/python

ERROR: The executable /home/vagrant/lib/azure-cli/bin/python3 is not functioning

ERROR: It thinks sys.prefix is '/u01/app/miniconda3/envs/azcli' (should be '/home/vagrant/lib/azure-cli')

ERROR: virtualenv is not compatible with this system or executable

Traceback (most recent call last):

  File "/tmp/azure_cli_install_tmp_zF6J3v", line 415, in <module>

    main()

  File "/tmp/azure_cli_install_tmp_zF6J3v", line 399, in main

    create_virtualenv(tmp_dir, install_dir)

  File "/tmp/azure_cli_install_tmp_zF6J3v", line 144, in create_virtualenv

    exec_command(cmd, cwd=working_dir)

  File "/tmp/azure_cli_install_tmp_zF6J3v", line 105, in exec_command

    subprocess.check_call(command_list, cwd=cwd, env=env)

  File "/u01/app/miniconda3/envs/azcli/lib/python3.11/subprocess.py", line 413, in check_call

    raise CalledProcessError(retcode, cmd)

subprocess.CalledProcessError: Command '['/u01/app/miniconda3/envs/azcli/bin/python3', 'virtualenv.py', '--python', '/u01/app/miniconda3/envs/azcli/bin/python3', '/home/vagrant/lib/azure-cli']' returned non-zero exit status 100.

(azcli) [vagrant@localhost azureinstall]$

Reference to fix the issue: https://github.com/Azure/azure-cli/issues

Reading 1: https://github.com/Azure/azure-cli/issues/26590

Key callout: 

I believe you are following https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=script, 

but this script is also almost "deprecated". You may use other approaches to install Azure CLI:

Use a native package manager, such as apt, dnf and tdnf: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli

Manually create a virtual environment with built-in library venv and install with pip: [Doc] Add doc for installing Azure CLI with pip #20476

Option 2: Using python directly perform what install.py attempted but without virtual environment instruction

https://github.com/Azure/azure-cli/issues/20476

Steps for windows:

# Create a virtual environment

python -m venv azure-cli-env

# Update pip

azure-cli-env\Scripts\python.exe -m pip install --upgrade pip

# Install azure-cli

azure-cli-env\Scripts\python.exe -m pip install azure-cli

# Run any Azure CLI commands

azure-cli-env\Scripts\az --version

How did we do on OL8:

1) # Create a virtual environment

#python -m venv azure-cli-env  ### skipped since I already have a virtual environment with conda

2) # Update pip

#azure-cli-env\Scripts\python.exe -m pip install --upgrade pip ### I installed the latest python and dependent package using conda

3) # Install azure-cli

azure-cli-env\Scripts\python.exe -m pip install azure-cli

Actual command:

python3 -m pip install azure-cli

Actual output:

(azcli) [vagrant@localhost ~]$ python3 -m pip install azure-cli

Collecting azure-cli

  Obtaining dependency information for azure-cli from https://files.pythonhosted.org/packages/4f/45/370fe2058345d81acf7bc1a23af95888e92104adbffa1bbc9e9403b2a769/azure_cli-2.52.0-py3-none-any.whl.metadata

  Downloading azure_cli-2.52.0-py3-none-any.whl.metadata (8.3 kB)

Collecting antlr4-python3-runtime~=4.9.3 (from azure-cli)

  Downloading antlr4-python3-runtime-4.9.3.tar.gz (117 kB)

     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.0/117.0 kB 4.7 MB/s eta 0:00:00

  Preparing metadata (setup.py) ... done

Collecting azure-appconfiguration~=1.1.1 (from azure-cli)

...

ccesscontrol-0.5.0 azure-synapse-artifacts-0.17.0 azure-synapse-managedprivateendpoints-0.4.0 azure-synapse-spark-0.2.0 bcrypt-4.0.1 certifi-2023.7.22 cffi-1.15.1 chardet-3.0.4 charset-normalizer-3.2.0 colorama-0.4.6 cryptography-41.0.3 deprecated-1.2.14 distro-1.8.0 fabric-2.7.1 humanfriendly-10.0 idna-3.4 invoke-1.7.3 isodate-0.6.1 javaproperties-0.5.2 jmespath-1.0.1 jsondiff-2.0.0 knack-0.11.0 msal-1.24.0b1 msal-extensions-1.0.0 msrest-0.7.1 msrestazure-0.6.4 oauthlib-3.2.2 packaging-23.1 paramiko-3.3.1 pathlib2-2.3.7.post1 pkginfo-1.9.6 portalocker-2.7.0 psutil-5.9.5 pycparser-2.21 pygments-2.16.1 pyopenssl-23.2.0 python-dateutil-2.8.2 pyyaml-6.0.1 requests-2.31.0 requests-oauthlib-1.3.1 scp-0.13.6 semver-2.13.0 six-1.16.0 sshtunnel-0.1.5 tabulate-0.9.0 typing-extensions-4.7.1 urllib3-2.0.4 websocket-client-1.3.3 wrapt-1.15.0 xmltodict-0.13.0

(azcli) [vagrant@localhost ~]$

4) # Run any Azure CLI commands

azure-cli-env\Scripts\az --version

Actual command:

az --version

(azcli) [vagrant@localhost ~]$ az --version

azure-cli                         2.52.0

core                              2.52.0

telemetry                          1.1.0

Dependencies:

msal                            1.24.0b1

azure-mgmt-resource             23.1.0b2

Python location '/u01/app/miniconda3/envs/azcli/bin/python3'

Extensions directory '/home/vagrant/.azure/cliextensions'

Python (Linux) 3.11.4 (main, Jul  5 2023, 13:45:01) [GCC 11.2.0]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

(azcli) [vagrant@localhost ~]$

5) Where is my az command installed

(azcli) [vagrant@localhost ~]$ which az

/u01/app/miniconda3/envs/azcli/bin/az

(azcli) [vagrant@localhost ~]$

So it is properly sitting in a subdirectory under my conda env.

(azcli) [vagrant@localhost ~]$ conda deactivate

(base) [vagrant@localhost ~]$

(base) [vagrant@localhost ~]$ az --version

-bash: az: command not found

(base) [vagrant@localhost ~]$

(base) [vagrant@localhost ~]$ conda activate azcli

(azcli) [vagrant@localhost ~]$ az --version

azure-cli                         2.52.0

core                              2.52.0

telemetry                          1.1.0

Dependencies:

msal                            1.24.0b1

azure-mgmt-resource             23.1.0b2

Python location '/u01/app/miniconda3/envs/azcli/bin/python3'

Extensions directory '/home/vagrant/.azure/cliextensions'

Python (Linux) 3.11.4 (main, Jul  5 2023, 13:45:01) [GCC 11.2.0]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

(azcli) [vagrant@localhost ~]$

YouTube Video: 

How to setup Conda, Create an environment with Python and Activate the environment

Agenda: How to setup Conda, Create an environment with Python and Activate the environment

Purpose: I needed different versions of python for different application, so I thought I use conda to manage that.

In this blog, we will see how to install conda and then how to create an environment with python 3.11& activate it as well.

Reference url: https://docs.conda.io/projects/miniconda/en/latest/

1) Setup Conda

Goto the VM
Navigate to the relevant directory

Steps from reference url:

mkdir -p ./miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ./miniconda3/miniconda.sh
bash ./miniconda3/miniconda.sh -b -u -p ./miniconda3
rm -rf ./miniconda3/miniconda.sh

./miniconda3/bin/conda init bash

#skip#~/miniconda3/bin/conda init zsh

Actual output:

[vagrant@localhost app]$ df -h .
Filesystem                           Size  Used Avail Use% Mounted on
/dev/mapper/appdata--vg-appdata--lv  3.9G   60M  3.8G   2% /u01
[vagrant@localhost app]$ pwd
/u01/app
[vagrant@localhost app]$ mkdir -p ./miniconda3
[vagrant@localhost app]$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ./miniconda3/miniconda.sh
--2023-09-03 12:50:54--  https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
Resolving repo.anaconda.com (repo.anaconda.com)... 104.16.131.3, 104.16.130.3, 2606:4700::6810:8303, ...
Connecting to repo.anaconda.com (repo.anaconda.com)|104.16.131.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 103219356 (98M) [application/x-sh]
Saving to: ‘./miniconda3/miniconda.sh’
./miniconda3/miniconda.sh      100%[==================================================>]  98.44M  8.61MB/s    in 11s
2023-09-03 12:51:06 (8.66 MB/s) - ‘./miniconda3/miniconda.sh’ saved [103219356/103219356]

[vagrant@localhost app]$ ls -altr
total 0
drwxr-xr-x. 3 vagrant vagrant 17 Sep  3 12:35 ..
drwxrwxr-x. 3 vagrant vagrant 24 Sep  3 12:50 .
drwxrwxr-x. 2 vagrant vagrant 26 Sep  3 12:50 miniconda3

[vagrant@localhost app]$ ls -altr miniconda3/
total 100804
-rw-rw-r--. 1 vagrant vagrant 103219356 Jul 13 19:01 miniconda.sh
drwxrwxr-x. 3 vagrant vagrant        24 Sep  3 12:50 ..
drwxrwxr-x. 2 vagrant vagrant        26 Sep  3 12:50 .

[vagrant@localhost app]$ bash ./miniconda3/miniconda.sh -b -u -p ./miniconda3
PREFIX=/u01/app/miniconda3
Unpacking payload ...
Installing base environment...

Downloading and Extracting Packages

Downloading and Extracting Packages
Preparing transaction: done
Executing transaction: done
installation finished.

[vagrant@localhost app]$ ls -altr
total 4
drwxr-xr-x.  3 vagrant vagrant   17 Sep  3 12:35 ..
drwxrwxr-x.  3 vagrant vagrant   24 Sep  3 12:50 .
drwxrwxr-x. 18 vagrant vagrant 4096 Sep  3 12:52 miniconda3

[vagrant@localhost app]$ ls -altr miniconda3
total 100876
-rw-r--r--.  1 vagrant vagrant     10732 Jul 13 18:43 LICENSE.txt
-rw-rw-r--.  1 vagrant vagrant 103219356 Jul 13 19:01 miniconda.sh
drwxrwxr-x.  3 vagrant vagrant        24 Sep  3 12:50 ..
drwxrwxr-x.  3 vagrant vagrant        17 Sep  3 12:51 x86_64-conda-linux-gnu
drwxrwxr-x.  3 vagrant vagrant        17 Sep  3 12:51 x86_64-conda_cos7-linux-gnu
drwxrwxr-x.  3 vagrant vagrant       146 Sep  3 12:52 ssl
drwxrwxr-x.  4 vagrant vagrant        30 Sep  3 12:52 man
drwxrwxr-x. 16 vagrant vagrant      4096 Sep  3 12:52 share
drwxrwxr-x.  2 vagrant vagrant      4096 Sep  3 12:52 sbin
drwxrwxr-x.  2 vagrant vagrant        30 Sep  3 12:52 compiler_compat
drwxrwxr-x. 26 vagrant vagrant      8192 Sep  3 12:52 include
drwxrwxr-x. 19 vagrant vagrant      8192 Sep  3 12:52 lib
drwxrwxr-x.  4 vagrant vagrant        35 Sep  3 12:52 etc
drwxrwxr-x.  3 vagrant vagrant        22 Sep  3 12:52 shell
drwxrwxr-x.  2 vagrant vagrant        19 Sep  3 12:52 condabin
drwxrwxr-x.  2 vagrant vagrant      4096 Sep  3 12:52 bin
drwxrwxr-x.  2 vagrant vagrant      4096 Sep  3 12:52 conda-meta
drwxrwxr-x. 73 vagrant vagrant      8192 Sep  3 12:52 pkgs
drwxrwxr-x.  2 vagrant vagrant         6 Sep  3 12:52 envs
drwxrwxr-x. 18 vagrant vagrant      4096 Sep  3 12:52 .

[vagrant@localhost app]$ rm -rf ./miniconda3/miniconda.sh
[vagrant@localhost app]$ pwd
/u01/app
[vagrant@localhost app]$ ./miniconda3/bin/conda init bash
no change     /u01/app/miniconda3/condabin/conda
no change     /u01/app/miniconda3/bin/conda
no change     /u01/app/miniconda3/bin/conda-env
no change     /u01/app/miniconda3/bin/activate
no change     /u01/app/miniconda3/bin/deactivate
no change     /u01/app/miniconda3/etc/profile.d/conda.sh
no change     /u01/app/miniconda3/etc/fish/conf.d/conda.fish
no change     /u01/app/miniconda3/shell/condabin/Conda.psm1
no change     /u01/app/miniconda3/shell/condabin/conda-hook.ps1
no change     /u01/app/miniconda3/lib/python3.11/site-packages/xontrib/conda.xsh
no change     /u01/app/miniconda3/etc/profile.d/conda.csh
modified      /home/vagrant/.bashrc
==> For changes to take effect, close and re-open your current shell. <==
[vagrant@localhost app]$ ls -altr /home/vagrant/.bashrc
-rw-r--r--. 1 vagrant vagrant 843 Sep  3 12:52 /home/vagrant/.bashrc
[vagrant@localhost app]$ cat /home/vagrant/.bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
# User specific environment
if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]
then
    PATH="$HOME/.local/bin:$HOME/bin:$PATH"
fi
export PATH
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
# User specific aliases and functions
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/u01/app/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/u01/app/miniconda3/etc/profile.d/conda.sh" ]; then
        . "/u01/app/miniconda3/etc/profile.d/conda.sh"
    else
        export PATH="/u01/app/miniconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<
[vagrant@localhost app]$

2) To create an environment:

reference url: https://docs.conda.io/projects/conda/en/stable/commands/create.html

>>> open a new putty terminal

conda create -n azcli python=3.11

Actual output:

(base) [vagrant@localhost ~]$ conda create -n azcli python=3.11
Collecting package metadata (current_repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 23.5.2
  latest version: 23.7.3
Please update conda by running
    $ conda update -n base -c defaults conda
Or to minimize the number of packages updated during conda update use
     conda install conda=23.7.3


## Package Plan ##
  environment location: /u01/app/miniconda3/envs/azcli
  added / updated specs:
    - python=3.11

The following packages will be downloaded:
    package                    |            build
    ---------------------------|-----------------
    openssl-3.0.10             |       h7f8727e_2         5.2 MB
    pip-23.2.1                 |  py311h06a4308_0         3.3 MB
    setuptools-68.0.0          |  py311h06a4308_0         1.2 MB
    ------------------------------------------------------------
                                           Total:         9.7 MB
The following NEW packages will be INSTALLED:
  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu
  bzip2              pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0
  ca-certificates    pkgs/main/linux-64::ca-certificates-2023.05.30-h06a4308_0
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1
  libffi             pkgs/main/linux-64::libffi-3.4.4-h6a678d5_0
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1
  libgomp            pkgs/main/linux-64::libgomp-11.2.0-h1234567_1
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1
  libuuid            pkgs/main/linux-64::libuuid-1.41.5-h5eee18b_0
  ncurses            pkgs/main/linux-64::ncurses-6.4-h6a678d5_0
  openssl            pkgs/main/linux-64::openssl-3.0.10-h7f8727e_2
  pip                pkgs/main/linux-64::pip-23.2.1-py311h06a4308_0
  python             pkgs/main/linux-64::python-3.11.4-h955ad1f_0
  readline           pkgs/main/linux-64::readline-8.2-h5eee18b_0
  setuptools         pkgs/main/linux-64::setuptools-68.0.0-py311h06a4308_0
  sqlite             pkgs/main/linux-64::sqlite-3.41.2-h5eee18b_0
  tk                 pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0
  tzdata             pkgs/main/noarch::tzdata-2023c-h04d1e81_0
  wheel              pkgs/main/linux-64::wheel-0.38.4-py311h06a4308_0
  xz                 pkgs/main/linux-64::xz-5.4.2-h5eee18b_0
  zlib               pkgs/main/linux-64::zlib-1.2.13-h5eee18b_0

Proceed ([y]/n)? y

Downloading and Extracting Packages
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate azcli
#
# To deactivate an active environment, use
#
#     $ conda deactivate
(base) [vagrant@localhost ~]$

3. To activate an environment:

conda activate azcli

Actual output:

(base) [vagrant@localhost ~]$ conda activate azcli
(azcli) [vagrant@localhost ~]$

(azcli) [vagrant@localhost ~]$ python --version
Python 3.11.4
(azcli) [vagrant@localhost ~]$

YouTube video: 

Thanks