Framework for Data Processing

Abstract

Until now, this book formed a basis for an approach to building intercultural social simulation: it described the purposes of this book and what approaches exist. Different scenarios of social interaction social simulation were introduced. Data that can be used in intercultural experiments was discussed. However, something is still missing, namely a robust framework that relies on these findings and implements flexible prototypes of social systems. Such framework would, for example, compose social systems that realize required simulation behavior and tackle shortcomings of existing approaches. This chapter describes the framework for statistical processing and prototyping, SocioFramework. Moreover, it presents additional findings focusing on intercultural processing.

Appendices

Appendix A: The HMM Classifier

This appendix shows an implementation of an HMM Classifier that relies on JAHMM and is compatible with WEKA:

public class JAHMM extends AbstractClassifier {




    //number of observations in case of emotional E/A

    //space - 5

    protected int m_NumClasses;




    //number of states in case of emotional E/A space - 5

    protected int m_States;




    //the number of the sequence attribute

    protected int m_SeqAttr = -1;




    //0 -- k-means, 1 -- Baum-Welch, 2 -- scaled Baum-Welch

    protected int m_LearningMethod = 0;




    //Builds the current classifier. m_SeqAttr specifies

    //the sequential attribute.

    @Override

    public void buildClassifier(Instances data) throws

    Exception {




        ...




        //build HMM

        OpdfIntegerFactory factory =

          new OpdfIntegerFactory(m_NumClasses);

        Hmm<ObservationInteger> hmm =

          new Hmm<ObservationInteger>(m_States, factory);

        hmm.getOpdf(0).fit(new ObservationInteger(4));




        List<List<ObservationInteger>> sequences;

        sequences = extractSequences(data);




        switch (m_LearningMethod) {

            case 0:

                // KMeans learning

                KMeansLearner<ObservationInteger> kml =

                   new KMeansLearner<ObservationInteger>

                   (m_NumClasses, factory, sequences);




                learntHmm_ = kml.learn();




                break;

            case 1:

                /* Baum-Welch learning */

                BaumWelchLearner bwl = new BaumWelchLearner();




                learntHmm_ = bwl.learn(hmm, sequences);




                break;




            case 2:

                BaumWelchScaledLearner bwsl =

                  new BaumWelchScaledLearner();




                learntHmm_ = bwsl.learn(hmm, sequences);




                break;

        }

    }




    //Classifies the given test instance

    @Override

    public double classifyInstance(Instance instance) throws

    Exception {




        Instances seq = instance.relationalValue(m_SeqAttr);




        //extracts a training sequence from the given instance

        List<ObservationInteger> sequence =

          extractSequenceFromInstance(seq.instance(0));




        int[] states =

          learntHmm_.mostLikelyStateSequence(sequence);




        double bestClass = states[states.length - 1];




        return bestClass;

    }

} //endclass

Appendix B: The ARFF Wrapper

This appendix presents the Java source of a wrapper that maintains a WEKA-compatible classifier and ARFF data in the prototypes of SS systems. Note that the wrapper uses names of the dataset files to extract and evaluate statistical features for identifying necessary POS processing and lemmatization done by TreeTagger (Schmid 1994). For example, TreeTagger assumes that POS tagging is necessary for evaluating features in a dataset with the name containing string _grammar_:

package coreEmotionalEngine.emotext;




//imports emotext

//import TreeTagger

//imports java

//imports WEKA




public class ARFFWrapper {




    //a base classifier used for analyzing texts

    private Classifier c_ = null;




    //name of the dataset file

    private String datasetName_ = null;




    //WEKA instances used for training and testing

    protected Instances instances_ = null;




    //interface to evaluate statistical features

    private IFeatureEvaluation fe_ = null;




    public Classifier buildClassifier(Classifier clsr,

      String datasetName) throws Exception {

        ...

        return classifier;

    }




    public ARFFWrapper(Classifier c, TreeTagger tagger,

      String datasetName) {

        treetagger_ = tagger;




        try {

            if (datasetName.contains("fusion") &&

               (datasetName.endsWith(".spec"))) {

              //build a fusion classifier

              ...

            else {

              c_ = buildClassifier(c, datasetName);

              datasetName_ = datasetName;

            }

        } catch (Exception e) {

           e.printStackTrace();

        }

    }




    public double classifyInstance(Instance instance) throws

      Exception {

        return c_.classifyInstance(instance);

    }




    public Instance buildInstance(String text) {

        Instance i = null;

        if (datasetName_.contains("_fusion")) {

            //a spec is found that identifies part datasets

            //used for fusion

            i = buildFusedInstance(text);

        } else if (datasetName_.contains("_lexical_")) {

            //build a lexical instance of processed data

            i = buildLexicalInstance(text);

        } else if (datasetName_.contains("_stylometry_")) {

            //build a stylometric instance of processed data

            i = buildStylometricInstance(text);

        } else if (datasetName_.contains("_grammar_")) {

            //build a grammatical instance of processed data

            i = buildGrammarInstance(text);

        } else if (datasetName_.contains("_deixis_")) {

            //build a deictic instance of processed data

            i = buildDeixisInstance(text);

        } else {

            assert (false);

        }

        return i;

    }







        private Instance buildFusedInstance(String text) {

        ...

        return instance;

        }




    private Instance buildDeixisInstance(String text) {

        ...

        return instance;

    }




    private Instance buildGrammarInstance(String text) {

        ...

        return instance;

    }




    private Instance buildLexicalInstance(String text) {

        ...

        return instance;

    }




    private Instance buildStylometricInstance(String text) {

      ...

      return instance;

    }

}

To build statistical instances, the ARFFWrapper class references variable fe_ that refers to the IFeatureEvaluation interface maintaining feature evaluation:

package coreEmotionalEngine.emotext;




public interface IFeatureEvaluation {

    public abstract double value(double original);

}

Three implementations of this interface are available: the presence evaluation that evaluates features according to their presence in the analyzed text as 1 or 0 (PresenceFeatureEvaluation), the inverse evaluation that evaluates a feature as a reciprocal frequency value (InverseFeatureEvaluation), or the frequency evaluation that evaluates a feature as a frequency value (FrequencyFeatureEvaluation). See (Osherenko 2011, p. 80) for details of feature evaluation.

Appendix C: Storing Configuration

This appendix shows a configuration file used to store the parameters of SocioFramework (&#x9; is interpreted by the XML engine in Java as a tab character; &#xa; as a linebreak; &quot; as a quotation mark):