Calculating the Market Share of US Companies with the Help of EDGAR

Edgar is classifying the reporting companies by SIC (Standard Industrial Classification) Code. We can use this information to calculate the total sales per sector and then calculate the % share of the individual company. This is helping us to identify the companies with a big market share. The result (which uses Spark) can be found in the following Gist. Alternatively here is a version which purely relies on Scala and Smart EDGAR

By pschatzmann, ago

Smile: Predicting the Direction of Stock Market Prices using a Random Forrest Classifier

In this demo we show how to forecast if the NASDAQ-100 is moving up or down. We do this with the help of a Random Forrest Classifier from the Smile Machine Learning Framework. I tried to replicate the result from a research paper authored by Luckyson Khaidem, Snehanshu Saha, Sudeepa Roy Dey: Data Preprocessing exponentially smoothing Features Relative Strength Index Stochastic Oscillator Williams %R Moving Average Convergence Divergence Price Rate of Change On Balance Volume Read more

By pschatzmann, ago

Smart-EDGAR is supporting Formulas now…

In the last Blog we demonstrated how we can calculate KPIs with the help of Spark. We have extended Smart Edgar functionality so that we can implement Calculated KPIs directly with the help of formulas. Here is a short demo in Scala which uses formulas the built in ‘coalesce’ method the built in ‘lag’ method the built in ‘percentChange’ method and displays the result in a BeakerX Jupyter Notebook as tables and charts using the Read more

By pschatzmann, ago

Calculating Financial KPIs with Scala, Spark and Smart-EDGAR¶

I am planning to use the Edgar data to determine and calculate some financial KPIs and feed these into a Neural Network. In my prior posts I described how to use Webservices to request and display Edgar information with the help of Python and Pandas. In the following Gist I show how we can directly use the ‘built in’ Java Query functionality of Smart-Edgar from Scala in order to calculated some financial KPIs.

By pschatzmann, ago

Processing 2.1 Mio Records from Solr in a Spark Cluster with BeakerX¶

I decided to build a repository of news headlines: I loaded all ‘New York Times’ headlines since the year 2000 and all Business related news from the ‘Guardian’ into the Solr Search engine. It has never been the intention to process all documents in one run but the goal was to search for the relevant articles with the help of the search engine and then process only the relevant headlines. Out of curiosity however, I Read more

By pschatzmann, ago

OpenNLP: Predicting Stock Movements from the News

In my last blog I demonstrated how to build a model that can predict if a stock is going up or down based on the news headlines using Spark MLLib. In this demo I will do the same – but with the help of OpenNLP. The solution consists of the following components OpenNLP (Text Classification) My News-Digest fuctionality (which I have described in my last blogs) Investor (Determination of the stock prices to calculate the labels ) confusion-matrix (to evaluate the Read more

By pschatzmann, ago

Processing 2.1 Mio Records from Solr in Scala

I decided to build a repository of news headlines: I loaded all ‘New York Times’ headlines since the year 2000 and all Business related News from the ‘Guardian’ into a Sorl Search engine. More details can be found in my prior blog. It has never been the intention to process all documents in one run but the goal was to search for the relevant articles with the help of the search engine and then process Read more

By pschatzmann, ago

Increasing the Solr Heap in Docker

Initially I used the Solr standard settings but with a big amout of data, I was running out of heap space. Fortunatly it is possible to define the heap space with the SOLR_HEAP environment variable. If you also don’t want to risk to loose your data you should also map the volume /opt/solr/server/solr/mycores Here is the docker-compose.yml file that I am using: version: ‘3.1’ services: solr: image: “solr:alpine” ports: – “8983:8983” volumes: – /srv/solr:/opt/solr/server/solr/mycores environment: Read more

By pschatzmann, ago