In the last Blog we demonstrated how we can calculate KPIs with the help of Spark. We have extended Smart Edgar functionality so that we can implement Calculated KPIs directly with the help of formulas. Here is a short demo in Scala which uses formulas the built in ‘coalesce’ method Read more…
I decided to build a repository of news headlines: I loaded all ‘New York Times’ headlines since the year 2000 and all Business related news from the ‘Guardian’ into the Solr Search engine. It has never been the intention to process all documents in one run but the goal was Read more…
In my last blog I demonstrated how to build a model that can predict if a stock is going up or down based on the news headlines using Spark MLLib. In this demo I will do the same – but with the help of OpenNLP. The solution consists of the following Read more…
In this blog we will demonstrate how we can predict if a stock is going up or down based on the news headlines. The solution consists of the following components: Spark MLLib (Machine Learning) My News-Digest (which I have described in my last blogs) Investor (we determine the stock prices Read more…
After we have seen that processing large amounts of data with Spark is efficient, we will demonstrate how we can use a Spark DataFrame to generate Charts with Vegas-Spark! We display the number of all new York Times Articles and the Business Guardian Articles over time.
I decided to build a repository of news headlines: I loaded all ‘New York Times’ headlines since the year 2000 and all Business related News from the ‘Guardian’ into a Sorl Search engine. More details can be found in my prior blog. It has never been the intention to process Read more…
Today I spent some time to figure out how to use Vegas (which is a plotting library for Scala) in Jupyter with the BeakerX Scala kernel. Here is the result!
Recently I have spent some time to investigate the options to access the history of news articles via an API. I was mainly interested in APIs which can be accessed free of charge. Here is the list of the most useful providers: Guardian – Easy API – Acceptable Rate Limits Read more…
I am planning to use the DL4J Doc2Vec implementation for a sentiment analysis. However, I don’t want to start with an empty network but the staring point should be a pre-trained network: The initial trining should be done with the Sentiment140 dataset which can be found at https://www.kaggle.com/kazanova/sentiment140. It contains Read more…
The basic goal of a ‘sentiment analysis’ is to classify a given text into positive, negative or neutral. SentiWordNet is a lexical resource for opinion mining. It assigns sentiments to each synset of WordNet which makes it possible to “calculate” an overall sentiment for a text. A SentiWordNet implementation can Read more…