Intense Analytics

Collaborative filtering is commonly used for recommender systems. These techniques aim to fill in the missing entries of a user-item association matrix. spark.ml currently supports model-based collaborative filtering, in which users and products are described by a small set of latent factors that can be used to predict missing entries. spark.ml uses the alternating least squares (ALS) algorithm to learn these latent factors. The implementation in spark.ml has the following parameters: numBlocks is the number of blocks the users and items will be partitioned into in order to parallelize computation (defaults to 10). rank is the number of latent factors in the model (defaults to 10). maxIter is the maximum number of iterations to run (defaults to 10). regParam specifies the regularization parameter in ALS (defaults to 1.0). implicitPrefs specifies whether to use the explicit feedback ALS variant or one adapted for implicit feedback data (defaults to false...

Like everything in engineering, there are tradeoffs to be made when picking these non-JVM languages for your Spark code. Java offers advantages like platform independence by running inside the JVM, self-contained packaging of code and its dependencies into JAR files, and higher performance since Spark itself runs in the JVM. If you chose to use Python, users lose such advantages. In particular, managing dependencies and making them available for PySpark jobs on a cluster can be a pain. In this blog post, I will explain what your options are. To determine what dependencies are required on the cluster ahead of time, it is important to understand where different parts of Spark code get executed and how computation is distributed on the cluster. Spark orchestrates its operations via the driver program. The driver program initializes a SparkContext , in which you define your data actions and transformations, e.g. map , flatMap , and filter . When the driver program is run, the Spark frame...

Event Loop  A programming construct that waits for and dispatches events or messages in a program. Traditional Muti-Threaded Model N Worker thread/process Each incoming connection handed to a worker, that worker/ thread is blocked until completion of operations. Examples: File I/O, DB I/O, Network I/O … Blocking  = Waste of resources NodeJS Event Loop Asynchronous Non-blocking I/O Instead of performing I/O in our code, we dispatch I/O events to Node’s event loop and let it handle process optimization, threads and concurrency Let’s review a typical example with 3 clients making calls to webserver and in turn to database. As, shown in image, with traditional multi-threaded system, there is a significant amount of blocking happening. On other-hand single threaded operation on NodeJS server runs asynchronously and efficiently. * This post uses excerpts from Wikipedia and Dr. Constantine Aaron Cois’s articles

Pivot4J Analytics is a recent replacement of JPivot OLAP Browser. Pivot4J Analytics can be used to consume SSAS Multidimensional Models. Setting up Pivot4J Analytics is very simple and should take less than 40 minutes. These are the steps: A. Configure HTTP access for your SSAS server Follow this TechNet article:  http://technet.microsoft.com/en-us/library/gg492140.aspx B. Install Recent JDK Download recent JDK from  https://jdk8.java.net/download.html And perform default install. C. Setup Environment Variable JAVA_HOME Open File Explorer Right Click on Computer and click on properties Click on Advanced Systems Setting Click on Advanced tab Click on Environment Variables Create a new system variable Enter the Variable name as JAVA_HOME and the value to your jdk bin path (In my case it is C:\Program Files\Java\jdk1.8.0\bin) NOTE Make sure u start with  ;  in the Value so that it doesn’t corrupt the other environment variables which is alread...

Learn how to create and execute MDX (Multidimensional Expressions) and DMX (Data Mining Extensions) stored procedures in SQL Server to perform advanced data analysis and business intelligence operations. Introduction to MDX and DMX SQL Server provides rich support for data analysis through its support of MDX and DMX languages. MDX is primarily used for querying and manipulating data in multidimensional databases, typically OLAP cubes. On the other hand, DMX is used for managing and querying data mining models, making it essential for advanced analytics and predictive modeling. In this post, we’ll explore how to write stored procedures that utilize these languages within SQL Server. A stored procedure allows you to automate your queries and can be used to encapsulate MDX or DMX scripts for reusable and efficient data analysis. Creating a Basic MDX Stored Procedure MDX queries are often used for complex rep...