Type Of Pretraining Research Articles

Toward a clinical text encoder: pretraining for clinical natural language processing with applications to substance misuse.

Our objective is to develop algorithms for encoding clinical text into representations that can be used for a variety of phenotyping tasks. Obtaining large datasets to take advantage of highly expressive deep learning methods is difficult in clinical natural language processing (NLP). We address this difficulty by pretraining a clinical text encoder on billing code data, which is typically available in abundance. We explore several neural encoder architectures and deploy the text representations obtained from these encoders in the context of clinical text classification tasks. While our ultimate goal is learning a universal clinical text encoder, we also experiment with training a phenotype-specific encoder. A universal encoder would be more practical, but a phenotype-specific encoder could perform better for a specific task. We successfully train several clinical text encoders, establish a new state-of-the-art on comorbidity data, and observe good performance gains on substance misuse data. We find that pretraining using billing codes is a promising research direction. The representations generated by this type of pretraining have universal properties, as they are highly beneficial for many phenotyping tasks. Phenotype-specific pretraining is a viable route for trading the generality of the pretrained encoder for better performance on a specific phenotyping task. We successfully applied our approach to many phenotyping tasks. We conclude by discussing potential limitations of our approach.

The Utilization of Verbal and Perceptual Cues by Preschool Children in Concept Identification Problems

OSLER, SONIA F.; DRAXL, MARILYNN; and MADDEN, JOHN. The Utilization of Verbal and Perceptual Cues by Preschool Children in Concept Identification Problems. CHILD DEVELOPMENT, 1977, 48, 1071-1074. The purpose of the experiment was to study the influence of perceptual and verbal pretraining upon concept attainment in the preschool age range. 96 preschool children were divided according to age into 2 groups. Following pretraining, each child received either a form or color concept in a stimulus set containing 1 relevant and 1 irrelevant dimension. The results showed that 4-year-olds were not influenced by any type of pretraining, while 5-year-olds profited from perceptual pretraining but failed to improve following verbal pretraining. The authors interpreted these findings as evidence for the primacy of perceptual encoding during ontogenetic development, consistent with the views of Piaget and Bruner, but in conflict with those who maintain that the capacity for verbal encoding precedes the ability for nonverbal encoding.

Effect of Level of Manifest Anxiety and Type of Pretraining on Timing Behavior

A great many studies have shown that subjects who differ in level of anxiety, as measured by the Manifest Anxiety Scale (Taylor, 1953), perform significantly differently on eyelid-conditioning and verbal-learning tests. There has, however, been no study investigating the effect of level of anxiety on complex operant conditioning. If it could be demonstrated that low anxiety and high anxiety have differential effects on operant conditioning, it would add generality to the Manifest Anxiety Scale as a selective device. To this end, high-anxiety and low-anxiety subjects were trained on a timing task with a schedule of differential reinforcement of low rate of responding (DRL). Furthermore, since animal studies show that performance on such a schedule is affected by the nature of pretraining (Schmaltz and Isaacson, 1966), the role of pretraining was also investigated. Briefly, the experimental design consisted of giving highand low-anxiety subjects either interfering pretraining (fixed-ratio schedule, which generates a high rate of responding), facilitatory pretraining (fixed-interval schedule, which generates a lower rate of responding with 'scallops'), or no pretraining.

Optional shift behavior of children as a function of age, type of pretraining, and stimulus salience

Optional shift behavior was studied within a factorial design involving four levels of age (nursery school children, first-, third-, and fifth-grade children), three types of pretraining (perceptual training, perceptual plus verbal labeling training, and a control condition), and two degrees of stimulus salience (objects vs. patterned stimulation). The perceptual and perceptual plus verbal pretraining procedures significantly and equally facilitated selection of optional reversal shift, and nursery children made fewer optional reversals than older Ss. Degree of stimulus salience as manipulated here had no effect on shift behavior. The findings are discussed in relation to perceptual, attentional, and verbal mediation theories of learning.

CHILDREN'S ACHIEVEMENT EXPECTATIONS AS A FUNCTION OF SOCIAL REINFORCEMENT, SEX OF S, AND TEST ANXIETY

24 8and 9-year-old children of each sex and at low and high levels of anxiety as measured by the Test Anxiety Scale for Children (N = 96) were divided equally into 6 experimental groups according to the type of pretraining (success, failure, or no pretraining experience) and reinforcement condition (social reinforcement or nonreinforcement from 1 of 2 males) in a subsequent angle-matching task. The Ss' achievement expectations were assessed prior to and following the experimental task. Major results were: (a) Following social reinforcement, expectations increased, the effect being stronger for girls than boys. (b) Following nonreinforcement, achievement expectations remained stable for both sexes. (c) Initial achievement expectations correlated negatively with test anxiety for girls. (d) Change in achievement expectation correlated positively with test anxiety for boys in the social reinforcement condition. The sex difference in the facilitating effects of social reinforcement on achievement expectations was considered as being part of a possible Sex of S by Sex of E interaction. It is suggested that sex differences in learning and cognition may be greater than has been recognized. Theorizing on the causes of the negative relation between a high level of test anxiety and children's achievement behaviors and intellectual

Social reinforcement as a function of test anxiety and success-failure experiences.

64 7-year-old children of each sex and at low and high levels of anxiety as measured by the Test Anxiety Scale for Children (N = 256) were divided equally into 4 experimental groups according to the type of pretraining (success or failure) and reinforcement condition (social reinforcement or nonreinforcement) in a subsequent marble-sorting task. Performance was higher under social reinforcement and lower under nonreinforcement for low-anxious Ss after success and high-anxious Ss after failure, but similar under the two reinforcement conditions and at an intermediate level for low-anxious Ss after failure and high-anxious Ss after success (p < .01). These results suggested a revision in the incentive

Group Performance Under Feedback that Confounds Responses of Group Members

more than one person, it is often true that the effect of his actions upon the environment is a net effect of the responses of several men. That is, the knowledge of results that is available to each group member is a single group output that combines in some fashion response measures of more than one individual. The manner in which the individual responses are combined into a single output measure may not be known to the group members, and a social psychologist studying the group may not even be able to specify how they are combined. In any case, each individual receives informative feedback that correlates imperfectly with his individual response. There is distortion in a group member's and the distortion should not be presumed simply to be random, either with respect to time or with respect to his own responses. The distortion is systematically related to the responses of his teammates and may in turn be related to his own responses which are generating a part of the feedback to his teammates. Under these conditions, which we shall call conditions of confounded feedback, there is no direct empirical evidence on the group performance curves and how they are affected by various training variables. The present paper reports an experimental study of the effects of two independent variables under conditions of socially confounded feedback in twoman groups. One independent variable is the type of pretraining, and the other is the relative weight accorded to the responses of the two teammates in the confounded feedback. In this study the combination of responses into a single output is a simple additive combination. There are two dependent variables: a measure of the accuracy of the group output and a measure of role differentiation (i.e., the extent to which members respond differently in contributing to a group output).