AI KR StrategistsThis plan defines the roles AI KR Strategists.Artificial Intelligence Knowledge Representation Community Group
AIKR CG
Organization_cd4a9bd6-0ec8-425c-ae47-9599f9b4b209Carl MattocksCoChairWork performed and works created for each AI value proposition is clearly and transparently documented and measured.Vision_861566c8-e9be-4642-b52f-f673fa499f4eTo be responsible and accountable for the selection, development, application and management of Knowledge Representation (KR) for Artificial Intelligence (AI).Mission_861566c8-e9be-4642-b52f-f673fa499f4eStrategic PlanDocument the vision, values, goals, objectives for one or more AIKR objectsGoal_995c0b60-d64c-445e-86c8-a6f755f5ed9a1An AI KR Object may be :
* an algorithm (example - enable an entity to determine consequences; a set of instructions that provide the ability to monitor and/or move the environment; the rules that are used to change/manipulate/interpret data)
* an ontology (which has a set of ontological commitments) See Goal - Ontological Statements (provides sufficient definition to allow measurement to be performed)
* an Intelligent Reasoning (fragmentary) Theory, such as,
* deduction,
* induction,
* abduction,
* by analogy,
* probabilistic,
* case-based
* a Reasoning Mechanism (computational environment), such as,
* natural language processer,
* rules engine,
* machine learning
* a Vocabulary (medium of human expression)Human-in-the-Loop ControllersHuman-in-the-Loop Controllers are humans that Train/Test AI systems.
They control the inputs of the humans when humans are in the loop.
As a simple example, they tag the initial images that are fed into the algorithms; later they come back and refine the patterns identified; ultimately they may also come back and test the outcome.
See:
* https://en.wikipedia.org/wiki/Human-in-the-loop
* https://humansintheloop.org/model-training/Model, test, evaluate and implement ethical approaches for Supervised Machine Learning of curated (labeled) data sets and the Active Learning training of algorithms via adjustment of parametersOntologyEmploy ontology content that removes ambiguity, supports performance measurement and enables buy inObjective_a3d0d024-38a2-44e2-8c05-feaf13fdfb1d1The AI Strategist will work, with the AI KR Strategist and / or other experts, to ensure that ontology content mitigates bias by employing a complete glossary of all the data used and utilizing an accurate representation of the (data) relationship rules identified in processing instructions. That is, the ontology, Representational Adequacy is able represent all the required knowledge;Inferential Adequacy is able to manipulate the knowledge represented to produce new knowledge (inferred from the original);Acquisitional Efficiency is able to acquire new knowledge using Human In-The-Loop and /or Human Out-of-The-Loop methodsAI KR StrategistsAI StrategistsAlgorithmsUnderstand the various different types of algorithms and where they can support business strategyObjective_72826a99-eaf8-482d-b07f-a09afa7d13792Algorithms' capabilities and limitations should be explained in a manner that is adapted to stakeholder concerns and identifies how accuracy, robustness, computational cost and stability will be measuredApplicationsUnderstand the potential applications of AI to business strategies.Goal_2a903634-050d-43e0-9d2e-f0b1f33286352RequirementsIdentify which areas of the requirements warrant AI solutions versus which can be achieved with other types of solutionsGoal_a4a5f259-af0b-47f7-8e34-6c8eaefdae193GlossariesEmploy definitions from one or more glossaries when explaining AIKR object audit data, veracity facts and (human, social and technology) risk mitigation factorsGoal_0083c58a-3d13-4e0e-95d1-8391c3f6414a4So that (business) people more readily understand the value that the glossaries bring.RisksIdentify and mitigate risks and known threatsGoal_bbeed24a-c843-427c-944e-08376a49ab9e5A guiding principle is that AIKR systems must mitigate risks.DARPAThis goal arose in reference to the DARPA initiative.ConsequencesIdentify and minimize adverse and/or unintended consequencesObjective_fab00957-6d05-461b-a684-197efdecef6e1"Environment" includes the natural environment, as well as socio-economic and societal environments.
* Minimise the risk of unintended consequences.
* AI shall do no harm
* When you're testing something, you should not alter the environmental conditions.
* The social and societal impact should be carefully consideredDataEnsure data quality and integrityObjective_545676f5-3cde-4aa5-9e22-1a4a0f108e852Data quality: the data is fit for its intended purpose/use. Is supported by a systematic method for driving agreement on the definitions of categories.
Data integrity: is the maintenance and assurance of the accuracy and consistency of data over its entire life-cycle. Is supported by a monitoring system that compares actual outcome with predicted accuracyBiasIdentify and reduce bias in AI KR objectsObjective_1ea3840a-0a5c-452e-afcf-1a486d38fbc53Bias is disproportionate weight in favour of or against an idea or thing, usually in a way that is closed-minded, prejudicial or unfair.
A bias is a systematic error.Security Guard against illegitimate access whilst ensuring legitimate accessObjective_036d44e3-15a7-416e-b93e-a6a9b79412294Security means protection as well as the measures taken to be safe or protected.ControlDesign the criteria to control the use (and misuse) of algorithms and dataObjective_0548440c-d869-4347-89de-6b8157947b6f5Control:
* control of the algorithms: To stop them from learning beyond our ability to control them.
* control of the people who develop (strategists and developers) and use algorithms: they can be used for good or for evil.AI KR StrategistsAI DevelopersUsers of AI systemsIntellectual PropertyManage Intellectual Property rights over AI KR worksObjective_703a1123-98b1-401c-a691-1cc441b4953b6What works can be protected, and what form of protection can be used for them?12Existing rightsNumber of worksPerformanceIndicator_a3e916c2-f559-499b-990d-c70667a5fe1dNumber of works with existing rights identified.13Created RightsUnit of worksPerformanceIndicator_8456a4c3-4628-4e18-a513-d2e7b55f37d9Number of works for which Intellectual Property rights have been created14Protected worksWorksPerformanceIndicator_dd25215a-04db-47da-bda1-1f57a84fdd3eProportion of works with intellectual property rights in place15Disputes raisedDisputesPerformanceIndicator_6daeccd3-ce51-4948-b5a2-015df706df67Number of disputes raised:
* against you, or
* by you
for innapropriate use of works16Disputes resolvedDisputesPerformanceIndicator_459e381e-9131-4fd8-a1f6-55f18bb7520cPrivacy Protect the rights of the individuals/corporations whose data is processedObjective_46348127-5596-4b21-8d17-5bf4517013167Ensuring data is processed with the permission of the people to whom it pertains.
E.g. GDPR, intellectual property, etc.GovernanceDesign governance in line with the risk toleranceObjective_5336be3a-69dd-40b8-b18f-fef6cb1e87a08Data and algorithm governance.ComplianceEnsure AI Systems comply with all applicable laws and regulations, such as, provision audit data defined by a governance operating modelGoal_b71896a0-3d86-4713-a720-15738315e36b6Compliance policies and procedures ensure that a planned change to a KR Object usage will comply with applicable laws/regulations during the identification, development, documentation, testing, validation, implementation, modification, use and retirement lifecycleEthicsEnsure AI Systems adhere to principles of ethicsGoal_bbcb3dc4-5946-4d7d-b43f-0a55af305cc27AutonomyFind the balance between human control/oversight and machine autonomyObjective_28520dbc-b02c-4e4f-a93e-e91ffaff06591Oversight controls will enable the assessment of algorithms, data and design processesVeracityVeracityObjective_7a9d8c77-e826-4a55-8641-e7812145de412AccountabilityAccountabilityObjective_fc02c4bf-a0cc-42b3-9452-05d06965e47f3ConfidentialityConfidentialityObjective_4fc0efcb-adaa-4ead-926b-4ab5512b62a54RobustnessEnsure AI Systems are designed to handle uncertainty and tolerate perturbation from a likely threat perspective, such as, design considerations incorporate human, social and technology risk factorsGoal_5a34fa22-8d74-402f-b111-d0e585de11a28OutcomesTrack AIKR object performance outcome via KPI (Key Performance Indicator) based on supervised learning models measurementsGoal_e2b04ebe-49d3-43f3-a723-a44135690f649Algorithm EvaluationEvaluate AlgorithmsGoal_56cd3982-542c-4719-965e-0bcce6606a0110Assess how well Algorithm results match actual outcomes to determine
* how sensitive inferences made are to the parameters and
* the proportion of observations made were accurately predicted.
When needed the algorithmic impact assessments will also identify cause and effect of any biases.Artificial Intelligence Knowledge Representation Community Group (AIKR CG)Community of InterestTrustworthinessAdvance use of AI safeguardsObjective_fa222026-9d57-4423-9433-9933bfe755e01Advance use of AI change management, knowledge representation performance evaluation, algorithmic impact assessment and context aware safeguards for a reliable, safe and transparent outcomeClassificationTrack Classification Performance IndicatorsObjective_964efa5e-58a7-4d9a-a839-daa8aef2a8572Ontological Statement: Classification Accuracy is the ratio of number of correct class label predictions to the total number of input samples data.
Ontological Statement: F1 Score measure the Harmonic Mean between precision and recall. The range for F1 Score is [0, 1]. It tells you how precise your classifier is (how many instances it classifies correctly), as well as how robust it is (it does not miss a significant number of instances).1Precision RecallPerformanceIndicator_25badc58-238a-4cb5-ad8e-c218b425b3a0Ontological Statement: Precision is the number of correct positive results divided by the number of positive results predicted by the classifier.
Ontological Statement: Recall is the number of correct positive results divided by the number of all relevant samples (all samples that should have been identified as positive).2AccuracyPerformanceIndicator_1611aab4-de88-4a4f-ad30-f74165037856Ontological Statement: Classification Rate or Accuracy is given by the relation: True Positives + True Negatives / All Instances (True & False Positives + True & False Negatives)3Confusion MatrixPerformanceIndicator_4a78f4f9-6bd5-4382-85c4-d0bfb0c16549Ontological Statement: A confusion matrix is a summary of prediction results on a classification problem. The number of correct and incorrect predictions are summarized with count values and broken down by each class (the types of errors being made)
Types :
* True Positives : The cases in which we predicted YES and the actual output was also YES.
* True Negatives : The cases in which we predicted NO and the actual output was NO.
* False Positives : The cases in which we predicted YES and the actual output was NO.
* False Negatives : The cases in which we predicted NO and the actual output was YES.
Accuracy for the matrix can be calculated by taking average of the values lying across the “main diagonal”
Type
StartDate
EndDate
Description
Target
Number of True Positives
Target
Number of False Positives
Target
Number of True Negatives
Target
Number of False Negatives
Actual
[To be determined]4Per-class accuracyPerformanceIndicator_6d27fb46-89e0-40ca-9fd6-680f760608bd5Log-LossPerformanceIndicator_102e78ab-4e9a-4d04-8476-06b7121b3294Ontological Statement: Logarithmic loss (related to cross-entropy) measures the performance of a classification model where the prediction input is a probability value between 0 and 1 - Log loss increases as the predicted probability diverges from the actual label
Logarithmic Loss or Log Loss, works by penalising the false classifications. It works well for multi-class classification. When working with Log Loss, the classifier must assign probability to each class for all the samples. where,
y_ij, indicates whether sample i belongs to class j or not
p_ij, indicates the probability of sample i belonging to class j
Log Loss has no upper bound and it exists on the range [0, ∞). Log Loss nearer to 0 indicates higher accuracy, whereas if the Log Loss is away from 0 then it indicates lower accuracy.
In general, minimising Log Loss gives greater accuracy for the classifier.6AUC-ROC CurvePerformanceIndicator_d784403b-241c-418c-bd14-7930f884a440Ontological Statement: check performance of multi - class classification AUROC (Area Under the Receiver Operating Characteristics) curve.Ontological Statement: Area Under Curve(AUC) is one of the most widely used metrics for evaluation. It is used for binary classification problem. AUC of a classifier is equal to the probability that the classifier will rank a randomly chosen positive example higher than a randomly chosen negative example.
True Positive Rate (Sensitivity) : True Positive Rate is defined as TP/ (FN+TP). True Positive Rate corresponds to the proportion of positive data points that are correctly considered as positive, with respect to all positive data points.
False Positive Rate (Specificity) : False Positive Rate is defined as FP / (FP+TN). False Positive Rate corresponds to the proportion of negative data points that are mistakenly considered as positive, with respect to all negative data points.7F-measurePerformanceIndicator_1621ab3f-2e83-484b-95cb-89b63ecb46d9F1 Score is the Harmonic Mean between precision and recall.
Ontological Statement: F-measure represents both Precision and Recall it helps to have a measurement that represents both of them. F-measure is calculated using Harmonic Mean (in place of Arithmetic Mean).
Ontological Statement: Mean Absolute Error is the average of the difference between the Original Values and the Predicted Values. It gives us the measure of how far the predictions were from the actual output.
Ontological Statement: Mean Squared Error(MSE) takes the average of the square of the difference between the original values and the predicted values.8NDCGPerformanceIndicator_2f9b2c7a-892f-4433-8705-00267505f2bcOntological Statement: Normalized discounted cumulative gain (DCG) is a measure of ranking quality. In information retrieval, DCG measures the usefulness, or gain, of a document based on its position in the result list.9Regression AnalysisPerformanceIndicator_8d8ced68-00f3-4604-a350-bab9e4984375Root Mean Square Error (RMSE)
Ontological Statement: Root Mean Square Error (RMSE) is the standard deviation of the residuals (prediction errors). Residuals are a measure of how far from the regression line data points are; RMSE is a measure of how spread out these residuals are.10Quantiles of ErrorsPerformanceIndicator_5e57c985-7a58-4cf6-b711-2cf7ad3ddd9eQuantiles (or percentiles), which is the element of a set that is larger than half of the set, and smaller than the other half.11"Almost correct" predictionsPerformanceIndicator_0ef5a0b6-499e-4128-a3fa-b112e098a49bKR ObjectsEvaluate KR Object PerformanceGoal_2fdd92f6-fce6-41f7-b914-993aac92123e11KR Object oversight mechanisms will define how performance measurements are used via human-in-the-loop, human-on-the-loop, and human-in-command approachesStrategyPlan_861566c8-e9be-4642-b52f-f673fa499f4e2020-04-012022-05-05Submitter_861566c8-e9be-4642-b52f-f673fa499f4eCarlMattocksCarlMattocks@WellnessIntelligence.Institute