当我尝试在 Azure 机器学习工作区中运行 Python 脚本进行数据预处理时,我无法访问 ADLS gen2 blob 容器中的数据。我正在开发 ML 管道,但似乎无法...
当我尝试在 Azure 机器学习工作区中运行用于数据预处理的 Python 脚本时,我无法访问 ADLS gen2 blob 容器中的数据。我正在开发 ML 管道,但似乎无法解决这个问题。
以下是我初始化凭证和 MLclient 的方法:
from azure.ai.ml import command
from azure.ai.ml import Input, Output
from azure.ai.ml import MLClient
from azure.ai.ml import Input, Output
from azure.identity import DefaultAzureCredential
import os
# Set up Azure ML client
credential = DefaultAzureCredential()
ml_client = MLClient(
credential=credential,
subscription_id="XXX",
resource_group_name="YYY",
workspace_name="ZZZ",
)
data_prep_job = command(
inputs={
"data": Input(type="uri_file", path=data_input_path),
"test_train_ratio": 0.2,
},
outputs={
"train_data": Output(type="uri_folder", path="azureml://datastores/workspaceblobstore/paths/output/train_data"),
"test_data": Output(type="uri_folder", path="azureml://datastores/workspaceblobstore/paths/output/test_data"),
},
code=data_prep_src_dir,
command="""python data_prep.py \
\--data ${{inputs.data}} --test_train_ratio ${{inputs.test_train_ratio}} \
\--train_data ${{outputs.train_data}} --test_data ${{outputs.test_data}} \
""",
environment="AzureML-sklearn-0.24-ubuntu18.04-py37-cpu:1",
display_name="Data Preparation Job",
compute="my-model-compute"
)
# Submit the job
submitted_job = ml_client.jobs.create_or_update(data_prep_job)
ml_client.jobs.stream(submitted_job.name)
我收到此错误:
Error Code: ScriptExecution.StreamAccess.Authentication Native Error: error in streaming from input data sources StreamError(PermissionDenied(None)) => permission denied when access stream. Reason: None PermissionDenied(None) Error Message: Authentication failed when trying to access the stream. Make sure you have correct permissions set up. Err("")| session_id=2fe2ca-a9c7-d0161be
当我尝试通过 DefaultCredentials() 通过 Azure ML 中的笔记本单元读取数据时,它工作正常。但是,当我尝试运行类似代码的 Python 脚本时,它给出了一个错误。我已为 AML 计算资源和我的工作区提供了 Blob 存储贡献者角色。也许,我没有正确传递身份验证信息?
当您使用 hline 绘制支撑和阻力数组时,我无法将数组转换为其中的整数值。出现以下错误:无法使用参数“price”=“call”op 调用“hline”...
当您使用 hline 绘制支撑和阻力数组时,我无法将数组转换为其中的整数值。
我遇到以下错误:无法使用参数“price”=“call 'operator SQBR' (array)”调用“hline”。使用了“array”类型的参数,但需要“input float”。
indicator("Supports and Resistances", overlay=true)
// Function to identify and return support and resistance levels
findSupportsAndResistances() =>
var supportLevels = array.new_float()
var resistanceLevels = array.new_float()
// Calculate support and resistance levels
for i = 4 to 50
low1 = low[i]
high1 = high[i]
// Find local min
if low1 < low[i-2] and low1 < low[i-3] and low1 < low[i-4] and low1 < low[i-5]
array.push(supportLevels, low1)
// Find local max
else if high1 > high[i-2] and high1 > high[i-3] and high1 > high[i-4] and high1 > high[i-5]
array.push(resistanceLevels, high1)
[supportLevels, resistanceLevels]
// Get support and resistance levels
[supports, resistances] = findSupportsAndResistances()
// Draw support and resistance levels
for i = 0 to array.size(supports) - 1
hline(supports[i], "Support " + str.tostring(i+1), color=color.green)
for i = 0 to array.size(resistances) - 1
hline(resistances[i], "Resistance " + str.tostring(i+1), color=color.red)
我尝试从数组中选择一个整数,供 hline 使用。
我们有一个内部构建和维护的数据平台。以下是问题的简要概述:该平台使用 AWS 服务。我们希望用户使用来自以下来源的数据创建 Power BI 仪表板...
我们有一个内部构建和维护的数据平台。以下是问题的简要概述:
我们探索了各种解决方案,唯一可行的似乎是使用 Azure VNet 数据网关。但是,这需要具有大量预留容量单位的高级 Power BI 许可证,成本很高。
有人有这方面的经验吗?我们可能错过了什么更简单的方法吗?任何帮助和指导都将不胜感激。
谢谢你!
以下是我们探索的内容:
p6
p7
p8
如果其中任何内容不准确,请纠正我们。
我正在尝试制作一个通用的 ARM 模板来部署虚拟机,它将接受一个对象参数,该参数将被传递给自定义脚本扩展并映射到
我正在尝试制作一个通用的 ARM 模板来部署虚拟机,它将接受一个对象参数,该参数将传递给自定义脚本扩展并映射到部署机器上的环境变量中。我想避免命名特定变量以最大限度地重复使用。在某些情况下,我还希望能够使用密钥库机密。
文档 描述了使用该 string
函数应将对象参数编码为 json 字符串,但是当我让脚本输出原始值时,它不是有效的 json,并且没有扩展密钥保管库引用,甚至没有枚举常规属性的值。 adminPassword 的常规密钥保管库引用完全按预期工作。
我的 ARM 模板如下所示:
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"languageVersion": "2.0",
"parameters": {
"adminPassword": {
"type": "secureString"
},
"envVars": {
"type": "object",
"defaultValue": null
}
}
然后我使用以下方法将值传递到脚本中 "commandToExecute": "[concat('powershell -ExecutionPolicy Unrestricted -File ', 'script-name.ps1', ' -envVarsJson ''', string(parameters('envVars')), '''')]"
以及参数文件值:
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"adminPassword": {
"reference": {
"keyVault": {
"id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-rg/providers/Microsoft.KeyVault/vaults/my-vault"
},
"secretName": "admin-password"
}
},
"envVars": {
"value": {
"apiKey": {
"reference": {
"keyVault": {
"id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-rg/providers/Microsoft.KeyVault/vaults/my-vault"
},
"secretName": "api-key"
}
},
"otherValue":{
"value": "Hello World!"
}
}
}
}
脚本接收的值如下所示: '{apiKey:{reference:{keyVault:{id:/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-rg/providers/Microsoft.KeyVault/vaults/my-vault},secretName:api-key}},otherValue:{value:Hello World!}}'
使用 ARM 模板对象作为包含密钥保管库引用的参数
我尝试将我们处理密钥库引用的方式直接更改为 arm 模板 ,除了这种方法之外,我还尝试将它们转换为环境变量。
azuredeploy.json
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"vmName": {
"type": "string",
"defaultValue": "vksbVM",
},
"adminUsername": {
"type": "string",
},
"adminPassword": {
"type": "securestring",
},
"customParameters": {
"type": "object",
}
},
"resources": [
{
"type": "Microsoft.Network/virtualNetworks",
"apiVersion": "2021-05-01",
"name": "[concat(parameters('vmName'), '-vnet')]",
"location": "[resourceGroup().location]",
"properties": {
"addressSpace": {
"addressPrefixes": [
"10.0.0.0/16"
]
},
"subnets": [
{
"name": "default",
"properties": {
"addressPrefix": "10.0.0.0/24"
}
}
]
}
},
{
"type": "Microsoft.Network/networkInterfaces",
"apiVersion": "2021-05-01",
"name": "[concat(parameters('vmName'), '-nic')]",
"location": "[resourceGroup().location]",
"dependsOn": [
"[resourceId('Microsoft.Network/virtualNetworks', concat(parameters('vmName'), '-vnet'))]"
],
"properties": {
"ipConfigurations": [
{
"name": "ipconfig1",
"properties": {
"subnet": {
"id": "[resourceId('Microsoft.Network/virtualNetworks/subnets', concat(parameters('vmName'), '-vnet'), 'default')]"
},
"privateIPAllocationMethod": "Dynamic"
}
}
]
}
},
{
"type": "Microsoft.Compute/virtualMachines",
"apiVersion": "2021-07-01",
"location": "[resourceGroup().location]",
"name": "[parameters('vmName')]",
"dependsOn": [
"[resourceId('Microsoft.Network/networkInterfaces', concat(parameters('vmName'), '-nic'))]"
],
"properties": {
"hardwareProfile": {
"vmSize": "Standard_DS1_v2"
},
"osProfile": {
"computerName": "[parameters('vmName')]",
"adminUsername": "[parameters('adminUsername')]",
"adminPassword": "[parameters('adminPassword')]"
},
"storageProfile": {
"imageReference": {
"publisher": "Canonical",
"offer": "UbuntuServer",
"sku": "18.04-LTS",
"version": "latest"
},
"osDisk": {
"createOption": "FromImage"
}
},
"networkProfile": {
"networkInterfaces": [
{
"id": "[resourceId('Microsoft.Network/networkInterfaces', concat(parameters('vmName'), '-nic'))]"
}
]
}
}
},
{
"type": "Microsoft.Compute/virtualMachines/extensions",
"apiVersion": "2021-07-01",
"location": "[resourceGroup().location]",
"name": "[concat(parameters('vmName'), '/CustomScriptExtension')]",
"properties": {
"publisher": "Microsoft.Azure.Extensions",
"type": "CustomScript",
"typeHandlerVersion": "2.1",
"autoUpgradeMinorVersion": true,
"settings": {
"fileUris": [
"https://testssvksbstor.blob.core.windows.net/sample/samplepow.sh"
]
},
"protectedSettings": {
"commandToExecute": "[concat('bash samplepow.sh ''', string(parameters('customParameters')), '''')]"
}
},
"dependsOn": [
"[resourceId('Microsoft.Compute/virtualMachines', parameters('vmName'))]"
]
}
],
"outputs": {
"adminPassword": {
"type": "securestring",
"value": "[parameters('adminPassword')]"
}
}
}
参数.json:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"vmName": {
"value": "vksbVM"
},
"adminUsername": {
"value": "azureuser"
},
"adminPassword": {
"reference": {
"keyVault": {
"id": "/subscriptions/98bccad1-ca91-4b24-83d6-78dfd797ff89/resourceGroups/vinay-rg/providers/Microsoft.KeyVault/vaults/samplevaultvkd"
},
"secretName": "admin-password"
}
},
"customParameters": {
"value": {
"param1": "value1",
"param2": "value2"
}
}
}
}
示例pow.sh:
#!/bin/bash
customParamsJson=$1
echo "$customParamsJson" | jq -r 'to_entries | .[] | "export \(.key)=\(.value)"' > /tmp/set_env_vars.sh
source /tmp/set_env_vars.sh
env > /tmp/environment_vars.txt
我在这里使用 bash 来尝试配置 Linux VM
部署:
参考 :
带有模板的 Key Vault 机密 - Azure 资源管理器 | Microsoft Learn
jq 手册: https://jqlang.github.io/jq/manual/
https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-linux