Model-Agnostic Assurance (ALSP)

Abstract

State-of-the-art AI assurance is often model- and domain-specific. We present two model-agnostic pipelines—Adversarial Logging Scoring Pipeline (ALSP) and Requirements Feedback Scoring Pipeline (RFSP)—that score explainability, safety, security, fairness, trustworthiness, and ethics. ALSP uses game-theoretic weighting, adversarial logging, and secret inversion to detect malicious inputs and quantify assurance. RFSP is user-driven: it gathers assurance weight preferences, segments data, and optimizes hyper-parameters (grid and Bayesian) to reflect desired goals. Both pipelines are validated on SCADA, telco, water, and telecom datasets, producing quantifiable assurance scores and surfacing trade-offs among AI goals.

Highlights

Unified assurance

Combines XAI, fairness, and security signals into a single actionable index for each prediction.

Game-theoretic weights

Shapley-based weighting of indices prioritizes features and outcomes that most impact assurance.

Adversarial logging

Tracks adversarial examples and secret inversion attempts to flag drift and malicious behavior.

Production-friendly

Model-agnostic hooks for tabular and vision models with per-sample reporting.

Approach

ALSP (model-driven). Three algorithms: (1) Weight Assessment multiplies Shapley values with domain-provided assurance labels to yield per-sample scores; (2) Reverse Learning logs every boosting epoch to surface loss minima and drift; (3) Secret Inversion trains an autoencoder to detect adversarial data via reconstruction errors (SAI/CAI).

RFSP (user-driven). Three algorithms: (1) Economic Equilibrium collects user-weighted goals (must sum to 100); (2) Extreme Data Segmentation builds a dedicated AIA set and maps statistical measures to assurance goals; (3) Model Optimization tunes hyper-parameters via grid search and Bayesian optimization, reporting trust scores from F1 deltas.

Experiments & Results

Datasets. SCADA (critical water network, intrusion detection), Telco (churn/plan selection, bias tests), Pima diabetes (8 features, 768 samples), Bank loans (GBDT logging), and synthetic water/telco benchmarks for assurance stress tests.

Weight Assessment. Shapley-weighted AI assurance columns (AIAC) produce per-sample scores; injected Gaussian bias visibly shifts score distributions (Fig. Distributions & Histogram of indices).

Reverse Learning. Custom GBDT logging finds loss minima at epoch 13; epochs beyond that degrade accuracy and are pruned.

Secret Inversion. Autoencoder detects adversarial SCADA inputs; thresholding top 1% reconstruction error separates attack traffic with >91% accuracy.

RFSP. User-weighted goals (16.6 each) combined with statistical measures (ANOVA, Kendall, MI, Chi2, KS, outliers, bias/variance) to yield weighted AIA scores. Bayesian hyper-parameter search improves trust (TAI) over defaults and grid search.

Statistical measures mapped to assurance goals
Measure	Implication	Limitation	Related goals
ANOVA-F	Linear dependence	Numeric→categorical	XAI, TAI
Kendall	Nonlinear dependence	Numeric→categorical	XAI, TAI
Mutual Information	General dependence	Data-agnostic	XAI, TAI
Chi-squared	Category dependence	Cat→cat	XAI, TAI
KS test	Distribution distance	Numeric	SAI, CAI
Outlier rate	Points outside 3σ	Numeric	SAI, CAI
Bias	Prediction accuracy	All cols	FAI, EAI
Variance	Prediction stability	All cols	FAI, EAI

Intrusion detection via Secret Inversion (AE)
Test set	Accuracy	F1	Precision	Recall
SCADA-1	0.9466	0.7194	0.9438	0.5813
SCADA-2	0.9128	0.7182	0.9707	0.5700

CAI score vs injected bias (SCADA)
Bias injected	CAI score
3.33%	54.4
2.67%	52.7
2.00%	53.4
1.34%	57.3
0.67%	63.6
0%	71.5

LightGBM hyper-parameters (Telco)
Hyper-parameter	Default	Bayesian	Grid
Learning rate	0.10	0.05	0.07
Max depth	-1 (unlimited)	14	9
Bagging fraction	1.0	0.8	1.0
Number of trees	100	396	100
Number of leaves	31	25	12

TAI (F1) improvements
Model setup	Telco F1	SCADA F1
Default	0.767	0.802
Grid search	0.746	0.794
Bayesian opt.	0.772	0.842

Gallery

Key artifacts from ALSP: assurance pipeline diagram, adversarial probes, fairness diagnostics, and assurance index distributions.

ALSP pipeline: XAI + fairness + security into ALS

Framework overview and component flow

Adversarial examples and perturbations

Loss trends during assurance training

Assurance index distributions across samples

Histogram of ALS scores

Interactive slider for assurance thresholds

Fairness analysis: gender bias diagnostics

Assurance index architecture

Localization of assurance signals

Decision path / drift illustration