OpenNLP: Predicting Stock Movements from the News

In my last blog I demonstrated how to build a model that can predict if a stock is going up or down based on the news headlines using Spark MLLib. In this demo I will do the same – but with the help of OpenNLP. The solution consists of the following components OpenNLP (Text Classification) My News-Digest fuctionality (which I have described in my last blogs) Investor (Determination of the stock prices to calculate the labels ) confusion-matrix (to evaluate the Read more

By pschatzmann, ago

Processing 2.1 Mio Records from Solr in Scala

I decided to build a repository of news headlines: I loaded all ‘New York Times’ headlines since the year 2000 and all Business related News from the ‘Guardian’ into a Sorl Search engine. More details can be found in my prior blog. It has never been the intention to process all documents in one run but the goal was to search for the relevant articles with the help of the search engine and then process Read more

By pschatzmann, ago

Increasing the Solr Heap in Docker

Initially I used the Solr standard settings but with a big amout of data, I was running out of heap space. Fortunatly it is possible to define the heap space with the SOLR_HEAP environment variable. If you also don’t want to risk to loose your data you should also map the volume /opt/solr/server/solr/mycores Here is the docker-compose.yml file that I am using: version: ‘3.1’ services: solr: image: “solr:alpine” ports: – “8983:8983” volumes: – /srv/solr:/opt/solr/server/solr/mycores environment: Read more

By pschatzmann, ago

News-Digest: Accessing the History of News Headlines¶

Recently I have spent some time to investigate the options to access the history of news articles via an API. I was mainly interested in APIs which can be accessed free of charge. Here is the list of the most useful providers: Guardian – Easy API – Acceptable Rate Limits – Access to over 1,900,000 pieces of content – Free for non-commercial usage New York Times – Provides API to search and separate API to Read more

By pschatzmann, ago

DL4J Doc2Vec – Sentiment Analysis using Sentiment140

I am planning to use the DL4J Doc2Vec implementation for a sentiment analysis. However, I don’t want to start with an empty network but the staring point should be a pre-trained network: The initial trining should be done with the Sentiment140 dataset which can be found at https://www.kaggle.com/kazanova/sentiment140. It contains 1,600,000 tweets extracted using the twitter api. In this Gist I describe how to train and save a DL4J Doc2Vec. The serialized model is available Read more

By pschatzmann, ago

DL4J – Sentiment Analysis with SentiWordNet¶

The basic goal of a ‘sentiment analysis’ is to classify a given text into positive, negative or neutral. SentiWordNet is a lexical resource for opinion mining. It assigns sentiments to each synset of WordNet which makes it possible to “calculate” an overall sentiment for a text. A SentiWordNet implementation can be found in DL4J in the deeplearning4j-nlp-uima artifact. This demo has been implemented in Scala using a Jupyter BeakerX Notebook.

By pschatzmann, ago

Predicting the Direction of Stock Market Prices using a Random Forest Classifier

In this demo I will show how to predict if the closing price of Apple, General Electric and Samsung Electronics is moving up or down. We do this with the help of a Random Forest Classifier. I tried to replicate the result from a research paper authored by Luckyson Khaidem, Snehanshu Saha, Sudeepa Roy Dey which is describing the following concept: Data Preprocessing exponentially smoothing Features Relative Strength Index Stochastic Oscillator Williams %R Moving Average Convergence Read more

By pschatzmann, ago