Machine Learning Tools Action Set
Example 15.1 Cross Validation of a Forest Model
Cross Validation of a Forest Model
This section contains PROC CAS code.
Note: Input data must be in a CAS table that is accessible in your CAS session. This table has a two-level name; the first level is your CAS engine libref and the second level is the table name. You refer to this table in the CAS procedure by specifying only the second level. For more information about two-level names, see Chapter 3: Shared Concepts in SAS Visual Data Mining and Machine Learning 8.3: Procedures Guide. For more information about PROC CAS and programming in CASL, see Getting Started with CASL, SAS Cloud Analytic Services: CAS Procedure Programming Guide and Reference, and SAS Viya: System Programming Guide.
This example shows how to use the crossValidateML action to perform k-fold cross validation of a forest model. The following DATA step loads the JunkMail data set from the Sashelp library into a data table named mycas.JunkMail. These statements assume that the CAS engine libref is named mycas, but you can substitute any appropriately defined CAS engine libref.
data mycas.JunkMail;
set sashelp.JunkMail;
run;
The following PROC CAS code uses the crossValidateML action to automatically perform k-fold cross validation of a forest model that is trained on the JunkMail data table:
proc cas noqueue;
mltools.crossValidateML /
modelType='forest'
kFolds=3
seed=12345
casOut='junkmail_forest_score'
trainOptions={
table={
name='junkmail',
where='Class NE .'
},
inputs={
"Address", "Addresses", "All", "Bracket",
"Business", "CS", "CapAvg", "CapLong",
"CapTotal", "Conference", "Credit", "Data",
"Direct", "Dollar", "Edu", "Email",
"Exclamation", "Font", "Free", "George",
"HP", "HPL", "Internet", "Lab",
"Labs", "Mail", "Make", "Meeting",
"Money", "Order", "Original", "Our",
"Over", "PM", "Paren", "Parts",
"People", "Pound", "Project", "RE",
"Receive", "Remove", "Semicolon", "Table",
"Technology", "Telnet", "Will", "You",
"Your", "_000", "_85", "_415",
"_650", "_857", "_1999", "_3D"
},
target='Class',
nominals={
{name='Class'}
},
casOut={
name='forest_junkmail_model', replace=true
}
}
;
run;
quit;
The "Cross Validation Fit Statistics" table in Output 15.1.1 displays the fit statistics for each fold and the fit statistic values averaged across all folds.
Output 15.1.1: Cross Validation Fit Statistics Output
| Cross Validation Fit Statistics | ||||||
|---|---|---|---|---|---|---|
| Fold | Sum of Frequencies | Average Squared Error | Divisor for ASE | Root Average Squared Error | Misclassification (MCE) | Multi-Class Log Loss |
| Fold 1 | 1535.00 | 0.1312 | 1535.00 | 0.3622 | 0.1713 | 0.4306 |
| Fold 2 | 1533.00 | 0.1293 | 1533.00 | 0.3595 | 0.1650 | 0.4253 |
| Fold 3 | 1533.00 | 0.1318 | 1533.00 | 0.3630 | 0.1657 | 0.4310 |
| Average | 1533.67 | 0.1307 | 1533.67 | 0.3616 | 0.1674 | 0.4289 |
CrossValidation of Forest Model
This section contains Lua code for the analysis in the CASL version of this example, which contains details about the results.
Note: In order to run this code, the data that are described in the CASL version need to be loaded into CAS. One way to do this is to convert the junkmail data to the comma-separated-value (CSV) file junkmail.csv and then use the following code to load the CSV file into CAS:
s:loadtable{casLib="casuser", path="junkmail.csv"}
For more information about coding in Lua, see Getting Started with SAS Viya for Lua and SAS Viya: System Programming Guide.
The following Lua code uses the crossValidateML action to automatically perform k-fold cross validation of a forest model that is trained on the JunkMail data table:
result = s:mltools_crossValidateML{
modelType='forest',
kFolds=3,
seed=12345,
casOut='junkmail_forest_score',
trainOptions={
table={
name='junkmail',
where='Class NE .'
},
inputs={
'Address', 'Addresses', 'All', 'Bracket',
'Business', 'CS', 'CapAvg', 'CapLong',
'CapTotal', 'Conference', 'Credit', 'Data',
'Direct', 'Dollar', 'Edu', 'Email',
'Exclamation', 'Font', 'Free', 'George',
'HP', 'HPL', 'Internet', 'Lab',
'Labs', 'Mail', 'Make', 'Meeting',
'Money', 'Order', 'Original', 'Our',
'Over', 'PM', 'Paren', 'Parts',
'People', 'Pound', 'Project', 'RE',
'Receive', 'Remove', 'Semicolon', 'Table',
'Technology', 'Telnet', 'Will', 'You',
'Your', '_000', '_85', '_415',
'_650', '_857', '_1999', '_3D'
},
target='Class',
nominals={
{name='Class'}
},
casOut={
name='forest_junkmail_model', replace=true
}
}
}
The following command displays the results table that is produced by this action call:
print(result.CrossValidateMLFitStat)
For details about the results of this analysis, see the PROC CAS version of this example.
Cross-Validation of Forest Model
This section contains Python code for the analysis in the CASL version of this example, which contains details about the results.
Note: In order to run this code, the data that are described in the CASL version need to be loaded into CAS. One way to do this is to convert the junkmail data to the comma-separated-value (CSV) file junkmail.csv and then use the following code to load the CSV file into CAS:
s.upload_file('junkmail.csv')
For more information about coding in Python, see Getting Started with SAS Viya for Python and SAS Viya: System Programming Guide.
The following Python code uses the crossValidateML action to automatically perform k-fold cross validation of a forest model that is trained on the JunkMail data table:
result = s.mltools.crossValidateML(
modelType='forest',
kFolds=3,
seed=12345,
casOut='junkmail_forest_score',
trainOptions={
"table":{
"name":"junkmail",
"where":"Class NE ."
},
"inputs":{
"Address", "Addresses", "All", "Bracket",
"Business", "CS", "CapAvg", "CapLong",
"CapTotal", "Conference", "Credit", "Data",
"Direct", "Dollar", "Edu", "Email",
"Exclamation", "Font", "Free", "George",
"HP", "HPL", "Internet", "Lab",
"Labs", "Mail", "Make", "Meeting",
"Money", "Order", "Original", "Our",
"Over", "PM", "Paren", "Parts",
"People", "Pound", "Project", "RE",
"Receive", "Remove", "Semicolon", "Table",
"Technology", "Telnet", "Will", "You",
"Your", "_000", "_85", "_415",
"_650", "_857", "_1999", "_3D"
},
"target":"Class",
"nominals":{
"Class"
},
"casOut":{"name":"forest_junkmail_model", "replace":True}
}
)
The following command displays the results table that is produced by this action call:
print(result.CrossValidateMLFitStat)
For details about the results of this analysis, see the PROC CAS version of this example.
Cross-Validation of Forest Model
This example is not available for the R programming language.