AI KR StrategistsThis plan defines the roles AI KR Strategists.Artificial Intelligence Knowledge Representation Community Group
AIKR CG
Organization_cd4a9bd6-0ec8-425c-ae47-9599f9b4b209Carl MattocksCoChairFor all AI systems to have clearly and transparently documented goals and performance data showing that they are being achieved.Vision_861566c8-e9be-4642-b52f-f673fa499f4eThe mission of an AI Strategist is to define the purpose and goals of AI systems, as well as the KPIs by which we can determine if the system is meeting its goals.Mission_861566c8-e9be-4642-b52f-f673fa499f4eStrategic PlanDocument the vision, values, goals, objectives for one or more AIKR objectsGoal_995c0b60-d64c-445e-86c8-a6f755f5ed9a1An AI KR Object may be :
* an algorithm (example - enable an entity to determine consequences; a set of instructions that provide the ability to monitor and/or move the environment; the rules that are used to change/manipulate/interpret data)
* an ontology (which has a set of ontological commitments) See Goal - Ontological Statements (provides sufficient definition to allow measurement to be performed)
* an Intelligent Reasoning (fragmentary) Theory, such as,
* deduction,
* induction,
* abduction,
* by analogy,
* probabilistic,
* case-based
* a Reasoning Mechanism (computational environment), such as,
* natural language processer,
* rules engine,
* machine learning
* a Vocabulary (medium of human expression)OntologyDefine and/or select ontology content that describe things sufficiently (just enough) well to allow buy-in, measurement and verification to be performedObjective_a3d0d024-38a2-44e2-8c05-feaf13fdfb1d1The AI Strategist will work, with the AI KR Strategist and / or other experts, to ensure that ontology content includes a glossary of the data used and the (data) relationship rules identified in processing instructionsAI KR StrategistsAI StrategistsAlgorithmsUnderstand the various different types of algorithms and how they support business strategyObjective_72826a99-eaf8-482d-b07f-a09afa7d13792ApplicationsUnderstand the potential applications of AI to business strategies.Goal_2a903634-050d-43e0-9d2e-f0b1f33286352RequirementsIdentify which areas of the requirements warrant AI solutions versus which can be achieved with other types of solutionsGoal_a4a5f259-af0b-47f7-8e34-6c8eaefdae193Ontological StatementsEmploy ontological statements when explaining AIKR object audit data, veracity facts and (human, social and technology) risk mitigation factorsGoal_0083c58a-3d13-4e0e-95d1-8391c3f6414a4Mitigate RisksMitigate known risks and proven threatsGoal_bbeed24a-c843-427c-944e-08376a49ab9e5A guiding principle is that AIKR systems must mitigate risks linked to bias, security and controlDARPAUnnamed roleThis goal arose in reference to the DARPA initiative.EnvironmentMinimize environmental impacts.Objective_fab00957-6d05-461b-a684-197efdecef6e1EfficiencyEfficiencyObjective_9b634c35-a188-404a-9f91-9abee4f178ad2ComplianceEnsure AI Systems comply with all applicable laws and regulations, such as, provision audit data defined by a governance operating modelGoal_b71896a0-3d86-4713-a720-15738315e36b6EthicsEnsure AI Systems adhere to principles of ethicsGoal_bbcb3dc4-5946-4d7d-b43f-0a55af305cc27AutonomyAutonomyObjective_28520dbc-b02c-4e4f-a93e-e91ffaff06591VeracityVeracityObjective_7a9d8c77-e826-4a55-8641-e7812145de412AccountabilityAccountabilityObjective_fc02c4bf-a0cc-42b3-9452-05d06965e47f3ConfidentialityConfidentialityObjective_4fc0efcb-adaa-4ead-926b-4ab5512b62a54RobustnessEnsure AI Systems are designed to handle uncertainty and tolerate perturbation from a likely threat perspective, such as, design considerations incorporate human, social and technology risk factorsGoal_5a34fa22-8d74-402f-b111-d0e585de11a28OutcomesTrack AIKR object performance outcome via KPI (Key Performance Indicator) based on supervised learning models measurementsGoal_e2b04ebe-49d3-43f3-a723-a44135690f649Algorithm EvaluationEvaluate AlgorithmsGoal_56cd3982-542c-4719-965e-0bcce6606a0110assess how well Algorithm results match actual outcome, to determine how sensitive inferences made are to the parameters and the proportion of observations made were accurately predictedArtificial Intelligence Knowledge Representation Community Group (AIKR CG)Community of InterestTrustworthyProvide the foundation for a trustworthy AIKRObjective_fa222026-9d57-4423-9433-9933bfe755e01Evaluation metrics are tied to machine learning tasks. Perhaps the easiest metric to interpret is the percent of estimates that differ from the true value by no more than X%.TrackTrack Classification Performance IndicatorsObjective_964efa5e-58a7-4d9a-a839-daa8aef2a8572Ontological Statement: Classification Accuracy is the ratio of number of correct class label predictions to the total number of input samples data.
Ontological Statement: F1 Score measure the Harmonic Mean between precision and recall. The range for F1 Score is [0, 1]. It tells you how precise your classifier is (how many instances it classifies correctly), as well as how robust it is (it does not miss a significant number of instances).1Precision RecallPerformanceIndicator_25badc58-238a-4cb5-ad8e-c218b425b3a0Ontological Statement: Precision is the number of correct positive results divided by the number of positive results predicted by the classifier.
Ontological Statement: Recall is the number of correct positive results divided by the number of all relevant samples (all samples that should have been identified as positive).2AccuracyPerformanceIndicator_1611aab4-de88-4a4f-ad30-f74165037856Ontological Statement: Classification Rate or Accuracy is given by the relation: True Positives + True Negatives / All Instances (True & False Positives + True & False Negatives)3Confusion MatrixPerformanceIndicator_4a78f4f9-6bd5-4382-85c4-d0bfb0c16549Ontological Statement: A confusion matrix is a summary of prediction results on a classification problem. The number of correct and incorrect predictions are summarized with count values and broken down by each class (the types of errors being made)
Types :
* True Positives : The cases in which we predicted YES and the actual output was also YES.
* True Negatives : The cases in which we predicted NO and the actual output was NO.
* False Positives : The cases in which we predicted YES and the actual output was NO.
* False Negatives : The cases in which we predicted NO and the actual output was YES.
Accuracy for the matrix can be calculated by taking average of the values lying across the “main diagonal”
Type
StartDate
EndDate
Description
Target
Number of True Positives
Target
Number of False Positives
Target
Number of True Negatives
Target
Number of False Negatives
Actual
[To be determined]4Per-class accuracyPerformanceIndicator_6d27fb46-89e0-40ca-9fd6-680f760608bd5Log-LossPerformanceIndicator_102e78ab-4e9a-4d04-8476-06b7121b3294Ontological Statement: Logarithmic loss (related to cross-entropy) measures the performance of a classification model where the prediction input is a probability value between 0 and 1 - Log loss increases as the predicted probability diverges from the actual label
Logarithmic Loss or Log Loss, works by penalising the false classifications. It works well for multi-class classification. When working with Log Loss, the classifier must assign probability to each class for all the samples. where,
y_ij, indicates whether sample i belongs to class j or not
p_ij, indicates the probability of sample i belonging to class j
Log Loss has no upper bound and it exists on the range [0, ∞). Log Loss nearer to 0 indicates higher accuracy, whereas if the Log Loss is away from 0 then it indicates lower accuracy.
In general, minimising Log Loss gives greater accuracy for the classifier.6AUC-ROC CurvePerformanceIndicator_d784403b-241c-418c-bd14-7930f884a440Ontological Statement: check performance of multi - class classification AUROC (Area Under the Receiver Operating Characteristics) curve.Ontological Statement: Area Under Curve(AUC) is one of the most widely used metrics for evaluation. It is used for binary classification problem. AUC of a classifier is equal to the probability that the classifier will rank a randomly chosen positive example higher than a randomly chosen negative example.
True Positive Rate (Sensitivity) : True Positive Rate is defined as TP/ (FN+TP). True Positive Rate corresponds to the proportion of positive data points that are correctly considered as positive, with respect to all positive data points.
False Positive Rate (Specificity) : False Positive Rate is defined as FP / (FP+TN). False Positive Rate corresponds to the proportion of negative data points that are mistakenly considered as positive, with respect to all negative data points.7F-measurePerformanceIndicator_1621ab3f-2e83-484b-95cb-89b63ecb46d9F1 Score is the Harmonic Mean between precision and recall.
Ontological Statement: F-measure represents both Precision and Recall it helps to have a measurement that represents both of them. F-measure is calculated using Harmonic Mean (in place of Arithmetic Mean).
Ontological Statement: Mean Absolute Error is the average of the difference between the Original Values and the Predicted Values. It gives us the measure of how far the predictions were from the actual output.
Ontological Statement: Mean Squared Error(MSE) takes the average of the square of the difference between the original values and the predicted values.8NDCGPerformanceIndicator_2f9b2c7a-892f-4433-8705-00267505f2bcOntological Statement: Normalized discounted cumulative gain (DCG) is a measure of ranking quality. In information retrieval, DCG measures the usefulness, or gain, of a document based on its position in the result list.9Regression AnalysisPerformanceIndicator_8d8ced68-00f3-4604-a350-bab9e4984375Root Mean Square Error (RMSE)
Ontological Statement: Root Mean Square Error (RMSE) is the standard deviation of the residuals (prediction errors). Residuals are a measure of how far from the regression line data points are; RMSE is a measure of how spread out these residuals are.10Quantiles of ErrorsPerformanceIndicator_5e57c985-7a58-4cf6-b711-2cf7ad3ddd9eQuantiles (or percentiles), which is the element of a set that is larger than half of the set, and smaller than the other half.11"Almost correct" predictionsPerformanceIndicator_0ef5a0b6-499e-4128-a3fa-b112e098a49bStrategyPlan_861566c8-e9be-4642-b52f-f673fa499f4e2020-04-012020-07-04